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Executive  Summary 


In  the  step  community  the  development  of  Application  Protocols  (APs)  has  emerged  as  a key  technical 
methodology.  The  development  of  an  AP  is  a long  and  complex  process.  APs  are  the  implementable  portion  of 
STEP.  This  being  true,  it  is  vital  that  implementations  of  APs  are  correct. 

In  addition  to  an  AP  implementation  conforming  to  the  AP  specification,  one  would  hope  that  the  AP  is  also 
useful.  Requirements  traceabihty  addresses  the  issue  of  the  usefulness  of  STEP.  Do  the  APs  meet  the  intended 
requirements?  Furthermore,  does  the  AP  meet  those  requirements  in  a way  that  can  be  tested? 

This  workshop  has  proven  to  be  most  valuable  simply  by  getting  the  AP  developers  and  testing  community  together 
for  an  extended  j)eriod  of  time,  discussing  these  issues.  The  recommendations  produced  by  the  participants  and 
summarized  in  this  document  are  put  forward  to  the  STEP  community  primarily  in  the  effort  to  ensure  the 
usefulness  of  STEP. 
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Abstract 


During  the  course  of  the  ISO/EPO  meeting  held  in  Seattle,  Washington,  in  April  of  1992,  a one-day  workshop 
was  held  on  AP  validation  requirements  and  procedures.  Included  in  this  workshop  were  representatives  from  all 
approved  AP  projects  and  planning  projects.  The  goal  of  this  workshop  was  to  reach  consensus  on  AP  validation 
and  conformance  testing  requirements.  The  objectives  were  to  establish  qualification  criteria  for  AP  validation 
reports;  to  discuss  lessons  learned  from  initial  AP  validation  activities  (i.e.  AP  requirements  validation,  ARM 
validation,  and  AIM  validation);  to  discuss  the  proposed  improvements  to  the  AP  Development  and  Approval 
Guidelines;  to  examine  completeness  requirements  for  AP  conformance  testing;  and  to  discuss  the  relationships 
between  the  components  of  the  AP  documents. 

This  document  is  an  informal  record  of  the  proceedings,  including  the  call  for  participation;  the  objectives, 
expectations,  and  agenda  for  the  workshop;  transcripts  of  presentations;  a workshop  summary  and  resulting 
recommendations,  and  a list  of  attendees.  Presentations  included: 

Guidelines  on  Writing  Standards  within  STEP 
Nigel  Shaw,  British  Aerospace 
Status  of  AP  Methods  and  Documentation 
Mark  Palmer,  NIST 

Model  Quality  Criteria  and  Metrics  Status 
Roger  Stumps,  Boeing 
Deploying  the  Voice  of  the  Customer 
Kurudi  Muralidhar,  777  - Ann  Arbor 

What  Information  is  Required  in  APs  to  Ensure  Compatible  Information  Exchange? 

Jon  Aas,  PEGS  Ltd 

Common  Methods  for  PDES,  Inc. 

Steve  Ryan,  Core  Mechanical  Project,  PDES,  Inc. 

Developing  and  Validating  Marine  Industry  Application  Protocols 
(on  behalf  of  NIDDESC)  Kent  Reed.  NIST 

The  Roles  of  Mapping  Tables  and  Conformance  Test  Purposes  in  STEP  Application  Protocols 
Julian  Fowler,  CADDETC  (presented  by  Jon  Owen,  CADDETC) 
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Introduction 


During  the  course  of  the  joint  International  Organization  for  Standardization  (ISO),  IGES/PDES 
Organization  meeting  held  in  Seattle,  Washington,  in  April  of  1992,  a one-day  workshop  was  held  on  AP 
validation  requirements  and  procedures.  Included  in  this  workshop  were  representatives  from  all  approved  AP 
projects  and  plaiming  projects.  The  goal  of  this  workshop  was  to  reach  consensus  on  AP  validation  and 
conformance  testing  requirements.  The  objectives  were  to  establish  qualification  criteria  for  AP  validation  reports; 
to  discuss  lessons  learned  from  initial  AP  validation  activities  (i.e.  AP  requirements  validation,  ARM  validation, 
and  AIM  validation);  to  discuss  the  proposed  improvements  to  the  AP  Development  and  Approval  Guidelines;  to 
examine  completeness  requirements  for  AP  conformance  testing;  and  to  discuss  the  relationships  between  the 
components  of  the  AP  documents. 

Funding  for  the  preparation  on  the  workshop  was  provided  by  the  Department  of  Defense's  Computer-Aided 
Acquisition  and  Logistic  Support  (CALS)  Office.  The  work  described  in  this  document  is  funded  by  the  United 
States  Government  and  is  not  subject  to  copyright 
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Call  for  Participation 


To:  AP  Project  Leaders,  WG4/P1  members,  WG4/P5  members,  and  WG6  members 
From:  Mary  Mitchell,  WG4/P1  chair  and  Mark  Palmer,  WG4/P5  chair 


Joint  Qualification  & Validation/ 

AP  Guidelines  and  Framework 
AP  Validation  Workshop 

Announcement 


In  Oslo,  significant  interest  was  expressed  in  holding  a joint  technical  workshop  on  AP  validation  requirements 
and  procedures.  The  goal  of  this  workshop  is  to  reach  consensus  on  validation  and  conformance  requirements  for 
APs.  The  objectives  of  this  meeting  are: 

♦ establish  qualification  criteria  for  AP  validation  reports; 

♦ discuss  lessons  learned  from  initial  AP  validation 

activities  (i.e.  AP  requirements  validation,  ARM  validation,  and  AIM  validation); 

♦ discuss  the  proposed  improvements  to  the  AP 
Development  and  Approval  Guidelines; 

♦ examine  completeness  requirements  for  AP 
conformance  testing;  and 

♦ discuss  the  relationships  between  the  components 
of  the  AP  documents. 

There  is  a need  to  examine  the  relationships  and  traceability  between  AP  requirements,  ARMs,  AIMs,  ARM  to 
AIM  mappings  and  Conformance  Requirements.  There  is  also  a need  to  discuss  conformance  test  purposes  and 
abstract  test  suites. 


Call  For  Position  Papers  and 
Presentations 


Short  position  papers  (3  to  5 pages)  are  solicited.  In  addition  presentations  for  the  workshop  (not  to  exceed  15 
minutes)  are  requested.  Please  notify  Mary  Mitchell  by  April  1, 1992  if  you  intend  to  present  material  at  the 
workshop.  Please  email,  FAX,  or  mail  papers  or  presentations  to  Mary  Mitchell  by  April  6, 1992.  For  all  position 
papers  and  presentations  which  arrive  by  this  date,  copies  will  be  provided  to  all  participants  at  the  workshop.  Any 
presentations  and  papers  which  arrive  after  that  time  will  only  be  distributed  in  the  workshop  proceedings. 

The  workshop  will  be  held  at  the  ISO/IPO  meeting  in  Seattle  on  Monday  afternoon  and  Tuesday  morning,  April 
13-14.  All  approved  AP  projects  and  planning  projects  are  requested  to  provide  a representative.  The  convenors 
of  both  WG4  and  WG6  have  approved  this  workshop.  If  a representative  from  your  Application  Protocol  project 
cannot  attend,  please  notify  Mary  Mitchell  by  April  10. 

We  would  like  to  keep  the  attendance  limited  to  25  or  30  people  to  keep  the  discussion  focused.  For  this  reason,  all 
approved  AP  projects  are  asked  to  send  not  more  than  two  individuals  and  all  AP  planning  projects  are  asked  to 
send  only  one  individual. 
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Informal  proceedings  will  be  prepared  from  the  workshop.  Copies  of  the  list  of  attendees,  all  presentations, 
position  papers  and  workshop  results  will  be  distributed  to  the  attendees  within  a month  of  the  workshop.  The 
proceedings  will  be  available  from  the  NIST  IGES/PDES/STEP  office.  Building  220  Room  A127,  Gaithersbmg, 
Maryland  20899,  attention:  Melissa  Andrews.  In  addition,  the  results  of  the  workshop  will  be  forwarded  to  Jerry 
Weiss,  the  SC4/PMAG  chair,  for  the  Seattle  Meeting  Minutes. 

A preliminary  agenda  is  included  in  this  mailing.  Feel  free  to  submit  comments  on  the  agenda. 

Thank  you  for  your  attention  to  this  matter. 


Objectives 


To  inform  AP  project  leaders  of  the  status  of  AP  Guidelines  and  STEP  quality  methods. 


To  promote  agreement  on  the  minimum  requirements  for  AP  quality. 
To  identify  techniques  to  evaluate  AP  quality. 

Expectations 


Provide  a common  frame  of  reference  for  all  AP  projects  on  quality  methods. 

Produce  proceedings  with  the  status  of  AP  Qualification  methods  and  a roadmap  for  future  directions. 

Recommendations  from  each  working  group: 

♦ Traceability  of  AP  requirements 

♦ AP  documentation  - improvements  and 
minimum  requirements 

♦ AP  project  planning  and  management 

Agenda 


Day  1,  Monday  afternoon 

♦ Introduction  of  attendees 

♦ Statement  of  workshop  goals  and  objectives  (15  minutes) 

♦ Status  of  AP  Guidelines  and  AP  Qualification  Manual  (30  minutes) 

♦ Invited  Presentations  (90  minutes) 

Day  2,  Tuesday  morning 

♦ Working  Sessions  (2  hours) 

Traceability:  How  to  trace  requirements  to  AP  implementation? 
Documentation:  What  are  the  minimum  requirements? 

♦ AP  project  plarming  and  management 
Formulation  of  conclusions  (1  hour) 
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Presenters 


A total  of  eight  people  made  presentations. 

Guidelines  on  Writing  Standards  within  STEP 

Nigel  Shaw  British  Aerospace 

Status  of  AP  Methods  and  Documentation 
Mark  Palmer  NIST 

Model  Quality  Criteria  and  Metrics  Status 

Roger  Stumps  Boeing 

Deploying  the  Voice  of  the  Customer 
Kurudi  Muralidhar  Murali  ITI  - Ann  Arbor 

What  Information  is  Required  in  APs 
to  Ensure  Compatible  Information  Exchange? 

Jon  Aas  PEGS  Ltd 

Common  Methods  for  PDES,  Inc. 

Steve  Ryan  Core  Mechanical  Project,  PDES,  Inc. 

Developing  and  Validating  Marine  Industry  Application  Protocols 
Kent  Reed  MST  (on  behalf  of  NIDDESC) 


The  Roles  of  Mapping  Tables  and  Conformance  Test  Purposes 
in  STEP  Application  Protocols 
Julian  Fowler  CADDETC 
presented  by  Jon  Owen  CADDETC 
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Guidelines  on  Writing  Standards  within  STEP 

Nigel  Shaw  British  Aerospace 

Editing  Committee  Chair 


My  starting  point  is  the  review  meeting  held  to 
develop  a UK  consensus  on  the  Part  203  ballot 
comments.  A mrniber  of  comments  described  the  fact 
that  it  is  a difficult  document  to  comprehend.  It 
became  clear  that  for  all  APs,  STEP  developers  had 
quite  a way  to  go  in  turning  these  documents  into 
usable  standards.  The  concern,  apart  from  reader 
difficulty,  was  that  the  mapping  between  the 
application  requirements  in  the  ARM  and  how  they 
are  represented  in  the  AIM.  This  mapping  is 
extremely  difficult  to  present.  There  are  a lot  of 
complex  ideas  represented  with  little  explanation. 
There  is  also  a concern  over  the  shear  length  of  these 
documents.  There  was  little  information  in  an  AP 
that  led  you  through  what  an  AP  is  and  how  the 
various  components  of  the  document  relate  to  each 
other. 

Depending  on  your  perspective,  e.g.,  I am  an 
implementor  or  I am  a user  or  I am  a testing  person, 
to  some  extent  biases  the  way  you  want  to  go  through 
the  document  and  how  you  interpret  the  relationship 
between  the  different  sections.  Further,  there  were 
editorial  problems  like  the  definitions  were  not 
included  from  the  integrated  resource  (IR)  Parts  but 
the  entities  were.  In  other  cases,  examples  and  notes 
from  the  IR  documents  which  were  copied  into  the 
AP  turned  out  to  be  inappropriate  to  the  AP  and  so 
forth.  So  it  was  very  clear  that  we  needed  to  tighten 
up  considerably  the  guidelines  provided  for  APs. 

In  Oslo  our  first  attempt  was  made  to  accomplish 


this.  What  it  really  did  was  generate  issues  for 
debate  at  this  meeting.  I'll  try  to  touch  on  some  of 
the  things  that  we  were  covered  at  the  AP  Guidelines 
meeting  on  Saturday.  However,  the  improvements  to 
the  Supplemental  Directives  and  AP  Guidelines 
documents  are  far  from  final.  Mark  Palmer 
mentioned  the  concern  about  an  AP  does  something 
for  the  user,  it  has  a purpose.  Implicit  in  that  is  the 
user  shall  do  certain  things  to  comply  with  the 
standard.  At  the  moment,  APs  aren't  really  spelling 
that  out  very  clearly.  They  say  - "you  must  have  a 
conforming  system",  "this  is  what  you  must  do". 
What  these  documents  need  to  do  is  describe  the 
process  of  using  this  AP.  The  AP  document  needs  to 
provide  the  answers. 

Its  been  pointed  out  that  for  the  first  release  of  STEP, 
there  are  a very  limited  munber  of  things  that  an 
implementor  needs  to  do  to  conform  to  the  ISO 
10303  standards.  Those  participating  in  the  AP 
Guidelines  meeting  felt  it  was  important  to  state  in 
the  introduction  what  a particular  AP  is  intended  to 
do.  Furthermore,  it  should  state  the  industrial  needs 
that  led  to  the  Aps  development 

An  AP  also  needs  to  explain  what  an  AP  is.  The 
reader  needs  a road  map  to  lead  them  through  the 
interrelationship  between  the  components  of  the  AP. 
It  is  very  important  to  clarify  the  scope  with  respect 
to  what  kind  of  product  an  AP  document  and  the 
reader  is  dealing  with.  The  intent  of  the  Editing 
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Committee  is  to  include  boiler  plate  text  for  the 
introduction  and  almost  all  components  of  an  AP  to 
make  it  clearer  how  the  components  relate,  where 
the  requirements  are  specified  within  the  document 
and  on  whom  the  requirements  are  placed. 

The  Editing  Committee  is  looking  to  improve  and 
include  boiler  plate  text  and  other  pieces  of 
information  required  by  Part  Editors  in  the 
Supplementary  Directives.  This  work  will  be 
coordinated  with  the  Guidelines  for  AP  Development 
for  all  areas  of  the  Application  Protocol.  The 
inclusion  of  Application  Interpreted  Constructs 
(AIC)  in  the  AP  document  is  a major  open  issue, 
especially  firom  the  documentation  viewpoint,  that 
needs  to  be  resolved. 

STEP  needs  to  ensure  that  the  context  is  very  clearly 
stated.  The  type  of  products  the  AP  is  dealing 
should  be  is  clearly  established.  The  implementation 
matters  such  as  who  shall  do  what  and  how.  There 
is  a missing  detail  which  needs  to  be  addressed  in 
the  AP  specifications.  They  should  state  when  do  the 
requirements  get  imposed. 

As  a very  large  group,  the  participants  of  this 
woilcshop  are  trying  to  improve  the  quality  and 
quantity  of  standard  text  as  required  in  application 
protocol  documents.  As  Editing  Chair,  I want  to  get 
STEP  documents  consistent  I am  a firm  believCT  in 
the  idea  that  if  someone  has  read  one  AP  that  fact  is 
going  to  help  them  understand  the  next  one.  The 
Editing  Conunittee  will  to  be  providing  in  the 
Supplementary  Directives,  hopefully  by  the  end  of 
May,  a large  amount  of  the  required  text  for 
Application  Protocols. 


Question  and  answer  period. 


Main  Points 


Focus  is  on  how  to  make  AP  documents  into  usable 
standards. 

Mapping  from  application  requirements  in  the  ARM 
and  representing  them  in  the  AIM  is  extremely 
difficult 

Other  concerns  include  the  sheer  length,  the  lack  of 
description  of  how  the  various  AP  components  are 
related,  and  editorial  inconsistencies  across  and 
within  documents. 

Roadmap  needed  to  lead  reader  through  the 
components  of  the  AP  and  through  using  the 
document 
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MODEL  QUALITY  CRITERIA  AND  METRICS  STATUS 


Roger  Stumps  Boeing 


member  of  the  IPO  Dictionary  Methodology  Committee 

Last  week  the  IPO  Dictionary  Methodology 
Committee  held  a three  day  working  session  at 
Boeing  on  the  Model  Quality  Criteria  document  to 
make  sure  that  the  ideas  in  it  are  still  valid  with  the 
current  STEP  development  process. 

The  first  item  for  discussion  is  the  current  state  of 
the  Model  Quality  Criteria  document,  and  the 
changes  that  are  proposed  to  be  made  to  it  Next  I 
will  discuss  the  future  enhancements  to 
accommodate  refinements  in  the  STEP  development 
process.  Finally,  we  need  to  develop  correlations 
between  the  multiple  quality  metrics  and  to  extend 
the  process  maturity  and  address  work  going  on  in 
the  STEP  implementation  areas.  The  document 
name  will  be  changed  to  Multi-Quality  Criteria 
Metrics  to  add  the  idea  of  both  the  identification  of 
quality  criteria  and  a way  to  measure  this  criteria. 

The  objective  is  to  provide  a meaningful  measure  of 
what  a STEP  model  status  is. 

There  is  an  issue  of  what  categories  of  quality 
documents  are  needed  verses  what  type  of  document 
we  currently  have.  The  viewpoint  of  the  committee  is 
that  a series  of  documents  are  needed.  It  was 
proposed  that  the  Model  Quality  Criteria  document 
be  worked  on  to  identify  what  needs  to  be  measured 
and  what  additional  metrics  are  needed  for  them.  In 
addition,  a quality  procedure  document  is  needed  in 
order  to  apply  the  criteria.  A practices  document. 


like  the  AP  Qualification  Manual,  is  needed  with 
forms  to  be  filled  out  when  evaluating  a STEP 
model.  Each  document  should  have  a section  which 
specifically  focuses  on  a certain  type  of  model. 

Those  are  the  three  categories  of  documents  that  we 
are  looking  at  right  now. 

Our  committee  proposes  to  continue  working  on  the 
document  framework  the  quality  management 
methods  in  STEP  with  the  available  resources.  We 
plan  to  pull  together  an  outline  of  what  we  think  the 
overall  document  architecture  for  quality 
management  in  STEP  should  contain.  It  is  a place  to 
start  and  throw  darts  at 

Concerning  the  practices  and  procedures  categories, 
there  has  not  been  too  much  energy  on  the 
committee  to  actually  work  on  that  type  of  document 
at  this  time.  (These  categories  are  being  developed 
byWG4/Pl.) 

We  need  more  feedback  back  on  our  document  We 
have  been  getting  a lot  of  feedback  but  not  always  at 
the  optunal  time  for  us  due  to  time  constraints  and 
other  constraints  on  people.  The  practices 
documents  developed  by  WG4/P1  and  P5  have  been 
conting  back  into  the  committee  but  we  also  need 
joint  discussion  between  the  projects  on  the  new 
thoughts  coming  in  these  documents  and  better 
coordination.  We  see  our  committee  as  belonging  at 
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this  level  - producing  the  requirements  and  assisting 
in  their  deployment  by  WG4/P1.  There  are  also  the 
practices  conunittees  that  are  assessing  the 
documents  and  deploying  the  results.  We  do  not 
have  any  overlap  in  effort  however,  the  two 
committees  should  be  working  together.  We  think 
this  is  missing  because  of  time  constraints.  Once 
again,  feedback  is  very  important  to  keep  these 
documents  alive  and  woiidng  as  STEP  defines  and 
refines  its  methods. 

We  have  received  requests  for  our  project  to  develop 
examples  of  what  the  elements  of  the  STEP 
documents  should  look  like  to  help  the  model  owners 
build  in  quality  throughout  the  process  instead  of 
trying  to  improve  the  documents  at  the  end  of  the 
process.  We  are  thinking  of  adding  a section  to  the 
document  on  defects.  We  will  be  putting  together 
some  guidelines  and  examples  about  how  to  spot  and 
correct  some  of  these  type  of  defects  and  how  to 
prevent  them  from  occurring  in  the  modeling. 

Within  that  concept,  we  have  found  that  tho^e  is  a 
problem  with  qualification  criteria  QC  8,  model 
syntax  evaluation,  and  QC  9,  formal  semantic 
representation.  We  are  working  on  a frameworic  for 
them  to  see  how  they  work  together.  The  sections  on 
QC  8 and  QC  9 will  be  totally  rewritten  in  the  next 
version  of  the  Model  Quality  Criteria  document 

The  committee's  strategy  is  to  identify  and  categorize 
common  modeling  defects  and  to  build  tonplate 
solutions  to  common  modeling  problems.  WG4/P2 
used  this  approach  for  some  common  integration 
problems.  These  will  be  documented  in  modeling 
practices  guidelines,  such  as  the  Express  usage 


guidelines.  We  are  hoping  that  when  the  model 
owners  develop  their  models  and  run  into  problems, 
they  go  through  the  defect  listing.  If  they  do  not  find 
the  defect  and  a solution  that  matches  their 
construct,  they  should  be  encouraged  to  come  back  to 
the  dictionary  and  qualification  practices 
committees.  They  should  describe  what  they  are 
trying  to  model  and  propose  their  solution  good  or 
bad.  They  should  be  provided  with  help  to  determine 
if  there  is  a better  way  of  modeling  this.  The 
communication  with  the  model  owners  needs  to 
come  back  to  our  group  too  and  not  just  to 
qualifications  practices.  We  need  communication 
between  practices  and  model  owners  back  into  the 
requirements  for  the  metrics  and  criteria. 

The  time  available  for  woric  on  this  document  is 
during  weekly  conference  calls  that  will  start  next 
month.  If  anyone  is  interested  in  participating  in 
those,  let  me  or  anyone  else  on  the  committee  know. 
We  will  be  glad  to  keep  you  informed  of  when  the 
conference  calls  are.  We  are  also  planning  to  have 
some  intCTim  releases  between  now  and  the  next 
document  release.  The  version  13  Model  Quality 
Criteria  document  will  probably  be  available  in 
October. 

Question  and  answer  period. 


Main  Points: 
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Keep  the  "Model  Quality  Criteria"  current  with 
STEP  methods  and  the  development  process. 

Creating  a document  framework  for  quality 
management  methods  in  STEP. 

A series  of  documents  are  needed  covering  quality 
criteria  and  metrics,  procedures,  and  use  of  quality 
management  methods  in  STEP. 

Dictionary  committee  needs  more  feedbxk  and 
coordination  from  model  developers  and 
qualification  practices. 

Qualification  criteria  for  model  syntax  and  use  for 
semantic  representation  are  being  enhanced. 

Model  Criteria  document  will  incorporate  section  on 
common  modeling  defects  and  recommended 
solutions. 

Document  will  be  re-titled  "Multi-Quality  Criteria 
Metrics"  and  published  in  October  1992. 
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Status  of  AP  Methods  and  Documentation 


Mark  Palmer  NIST 

WG4,  P5  AP  Guidelines  and  Framework  Project  Leader 


Overview  of  AP  Progress 
To  establish  the  context  for  the  discussions  on  AP 
validation,  I want  to  go  over  the  activity  diagrams 
for  the  AP  Development  Process,  the  responsibilities 
of  an  AP  project  and  the  component  elements  of  an 
AP.  With  this  information  we  can  identify  where  AP 
validation  plays  a role  in  the  process.  From  a high 
level,  we  have  three  primary  steps  in  developing  an 
AP.  First,  plan  and  develop  the  AP.  Next,  develop 
its  abstract  test  suite.  Finally,  the  development  of  a 
prototype  AP  implementation  is  strongly  encouraged 
prior  to  the  release  of  the  AP  for  DIS  ballot 
Decomposing  the  plan  and  develop  (A41)  activity, 
we  have  five  basic  steps:  develop  scope  and 
requirements,  develop  the  Application  Reference 
Model,  develop  the  Application  Interpreted  Model, 
develop  conformance  requirements  and  test 
purposes,  and  then  complete  your  AP  document 

I will  review  the  current  stete  of  the  documents 
which  focus  on  Application  Protocols.  The  three 
documents  currently  being  worked  on  within  SC4 
are:  Guidelines  for  the  Development  and  Approval 
of  STEP  APs,  WG4/N34;  Issues  and 
Recommendations  for  a STEP  AP  Framework,  a 
NIST  document;  and  the  STEP  AP  Qualification 
Manual,  working  draft  .05.  The  AP  Qualification 
Manual  and  the  AP  Guidelines  for  the  Development 
of  APs  have  been  moving  forward  in  parallel.  There 
have  been  modMcations  to  both  to  keep  them 


synchronized.  There  is  work  being  done  on  the  next 
AP  Framework  document.  And  as  a result  of 
meetings  of  the  Qualification  Project  WG4/P1, 
there  is  a recognition  for  the  need  of  a Quality 
Management  Structure.  The  Qualification  Project  is 
trying  to  evolve  from  a state  of  inspection  at  the  end 
of  the  process  to  providing  the  necessary  tools  for 
quality  management  as  the  AP  components  are 
developed  incrementally.  The  AP  Guidelines 
describes  three  groups  of  components  in  an  AP  - 
Group  1,  with  the  scope,  requirements,  ARM,  and 
ARM  validation  report.  These  components  are 
qualified  prior  to  proceeding  into  the  development  of 
an  Application  Interpreted  Model  (AIM).  The  AIM 
is  developed  and  also  validated  and  the  elements  are 
qualified  as  Group  2 before  developing  conformance 
requirements  and  test  purposes.  The  conformance 
requirements,  test  purposes  and  the  completed  AP 
documentation  is  then  qualified  as  Group  3 prior  to 
submission  for  review  and  approval  by  the  SC4 
Editing  Committee. 

The  requirement  for  AP  validation  has  always  been 
recognized  but  not  fully  exercised  in  existing  AP 
projects.  AP  validation  is  an  identified  task  in  the 
development  of  an  AP  and  plays  a critical  role  in 
delivering  usable  APs  to  the  STEP  community. 

When  I come  to  ISO  and  IPO  meetings,  I try  to  have 
a current  list  of  AP  development  issues.  I have 
extracted  from  that  list  the  issues  that  are  applicable 
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to  AP  validation  so  you  have  some  issues  with  which 
to  start  these  discussions.  I will  briefly  go  through 
these  issues.  If  anyone  wishes  to  question  the  merit 
of  these  issues,  please  raise  your  concerns  at  this 
time. 

There  is  the  issue  of  verification  of  the  international 
consensus  on  the  AP  scope  and  requirements.  We 
have  a process  and  procedures  defined  for  that  task 
but  1 do  think  it  has  not  been  successfully  exercised 
in  all  arenas.  There  are  cases  of  overlapping  APs 
and  conflicts  in  what  are  the  industrial  requirements 
that  are  the  driver  for  any  AP.  Under  the  issue  of 
ARM  development  and  validation,  there  are  issues 
on  the  required  specificity  of  the  application 
semantics  and  the  required  level  of  detail  for  the 
ARM. 

Likewise,  there  is  still  work  being  done  on  the  rules 
for  defining  Units  of  Functionality.  How  does  one 
vCTify  the  validity  of  those  Units  of  Functionality 
within  the  domain  of  the  AP?  Tho-e  is  also  an  issue 
on  the  role  of  these  "functional  building  blocks"  in 
an  AP.  Some  of  the  APs  have  functional  levels  in 
their  APs,  i.e.  how  do  we  build  upon  a core  ARM  to 
provide  increasing  levels  of  functionality.  This  has 
ramifications  on  all  the  downstream  activities  that 
must  support  the  AP.  On  AIM  development  and 
validation,  there  is  a fundamental  issue  of  ensuring 
traceability  from  the  requirements  to  the  AIM  and 
eventually  to  the  ATS.  We  do  not  have  the  tools  and 
the  methods  in  place  to  ensure  comprehensive 
traceability.  Likewise,  we  have  the  issue  of  how  to 
prepare  the  EXPRESS  expanded  form  from  the  short 
form.  There  are  some  verification  issues  here  as  well. 
Likewise,  there  are  issues  on  how  to  assess  the  AIM 


and  the  need  to  define  procedures  for  building  and 
validating  AICs. 

Moving  on  to  test  purposes,  abstract  test  suites,  and 
executable  test  cases,  we  have  a similar  set  of  issues 
on  granularity  as  with  the  ARM.  There  are  also 
issues  with  the  viability  of  one  set  of  test  purposes 
being  useful  for  multiple  implementation  forms. 
Finally,  under  the  issue  of  what  I classify  as  AP 
documentation,  there  is  the  issue  of  expanding  the 
mapping  table.  The  initial  role  of  the  mapping  table 
was  just  to  identic  each  ARM  concept  and  how  the 
application  concept  would  be  represented  by  the 
AIM  construct  However  the  constraints  on  how  this 
AIM  construct  is  used  are  equally  important. 

One  of  the  insights  that  came  out  of  the  Oslo 
meeting  was  that  if  we  actually  expanded  that 
mapping  table  to  a sufficient  level  of  detail,  the 
mapping  table  could  provide  much  of  what  is  now 
required  in  the  test  purposes,  i.e.  one  would  identify 
each  unique  path  and  that  would  be  the  testable 
structure  for  clause  6 in  the  APs.  The  issue  is 
making  the  mapping  table  more  robust  and  more 
understandable  so  that  some  of  what  is  in  the 
mapping  table  does  not  need  to  be  repeated  in 
another  section  of  the  document.  The  issue  on 
enforcing  validation  and  the  documentation  of  the 
AP  validation  has  been  raised  already. 

Thou  are  also  issues  on  the  ability  to  obtain  all 
ARM  concepts  from  the  AIM  constructs,  i.e.,  to 
promote  completeness  by  generating  the  reverse 
mappings.  There  is  also  the  issue  of  the  contents 
from  the  AP  usage  guide.  Much  of  the  work  today 
has  been  looking  at  what  is  a compliant 
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implementation  of  an  AP.  There  is  a corresponding 
issue  of  how  does  one  ensure  compliant  data  sets?  It 
is  in  the  delivery  of  compliant  data  sets  that  the  AP 
usage  guide  may  finally  play  a critical  role  but  we 
have  not  clarified  that  issue.  This  has  been  a brief 
description  of  the  issues  which  confront  the 
development  of  validated  APs. 

Question  and  answer  period. 


Main  Points: 


Endorses  the  development  of  a Quality  Management 
Structure  for  STEP. 

The  requirement  for  AP  Validation  has  not  been 
exercised  by  all  existing  AP  projects  and  this  raises 
qu^ity  issues. 

There  are  many  unresolved  issues  with  respect  to  AP 
development  & validation. 

Qualification  is  evolving  firom  inspections  at  the  end 
of  the  process  to  incremental  evaluations  and  more 
direct  assistance  to  Part  Editors. 

Proposal  to  have  an  ISO  ballot  to  determine  if  there 
is  intemational  consensus  of  the  AP  scope  and 
requirements. 
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Deploying  the  Voice  of  the  Customer 

Kurudi  Muralidhar  Murali  ITI  -Ann  Arbor 


Member  of  WG6,  Conformance  Testing 

Tracking  Requirements  for  Applications 
One  of  the  methods  that  has  been  used  in  tracking 
requirements  in  industry  is  called  Quality  Function 
Deployment.  This  method  works  fine  for  tracking 
the  requirements  of  all  types.  In  addition,  it  provides 
for  the  prioritization  of  requirements  in  a structured 
approach.  This  technique  is  similar  to  other 
structured  requirements  approaches.  QFD  may  not 
be  the  best  technique  but  it  allows  you  to  track  and 
analyze  the  requirements,  QFD  provides  a problem 
solving  group  with  a system  for  developing  a 
conunon  understanding.  The  best  thing  we  found 
out  about  using  this  method  was  the  conunon  metrics 
it  provides.  The  teams  which  have  used  it  are 
comfortable  that  they  are  working  with  the  same 
meaning.  That  is  where  it  helped. 

There  are  various  business  needs  being  addressed  by 
STEP.  There  are  maybe  similar  business  needs 
between  particular  APs.  What  we  are  trying  to 
achieve  could  be  a functional  requirement; 
functionality  could  be  a business  need.  There  may  be 
several  reasons  for  achieving  this  particular 
requirement  so  we  can  track  them  with  QFD  and  we 
can  assign  an  importance  number  for  each  of  these 
needs.  The  first  step  in  the  process  is  identifying  the 
requirements,  defining  them,  and  then  collectively 
assigning  a priority  what  is  important  about  them. 

How  important  is  using  a formal  requirements 


definition  and  tracking  method?  A method  like  this 
helps  to  build  a consensus  among  groups  of  people 
and  can  help  you  to  finalize  what  are  the  best 
requirements.  What  are  the  highest  needs?  Based 
on  evaluations  of  the  various  approaches  for  a 
particular  solution  or  how  you  want  to  meet  the 
particular  requirement,  you  can  start  assigning  rules 
so  you  can  get  absolute  measure  of  importance. 

Later  in  the  process  when  you  have  actually  finished 
your  AP,  you  can  take  a particular  requirement  from 
applying  this  method  and  you  can  track  it  all  the  way 
through  and  see  how  far  the  requirement  will  really 
implement. 

To  provide  more  STEP  type  of  things  from  a general 
product  design,  we  used  a QFD  approach  for  the 
analysis  of  an  abstract  test  notation  for  the  abstract 
test  suites.  Abstract  test  notation  is  a language  use  to 
describe  abstract  test  cases  for  APs.  In  applying  this 
approach,  there  were  several  languages  that  were 
available  for  use  as  an  abstract  test  notation.  We 
looked  at  the  various  requirements  for  such  an 
abstract  test  notation.  It  should  have  some  type  of 
ability  to  manipulate  the  instances  of  product  data  so 
that  they  can  test  the  validation  of  an  AP.  There  is 
some  testing  related  administrative  information  like 
where  this  test  came  from.  QFD  was  used  to 
establish  the  high  level  requirements  for  the  notation 
for  specifying  abstract  test  cases.  This  is  important 
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so  that  we  can  prove  the  capabilities  that  a language 
has  matched  the  requirements  and  that  the  language 
has  something  we  want  We  also  assigned 
importance  to  each  particular  requirement  in  a 
collective  fashion.  QFD  helped  us  direct  our  focus  to 
the  features  that  a candidate  language  most  needed 
to  support 

Question  and  answer  period. 


Main  Points: 


STEP  should  consider  using  a formal  requirements 
method  to  ensure  traceability. 

QFD  provides  the  ability  to  trace  a requirement  from 
its  inception  through  the  deployment  of  a solution. 

QFD  was  used  for  the  capability  survey  of  potential 
abstract  test  notations. 

The  metrics  within  QFD  provided  a means  of 
measuring  the  relative  advantages  of  each  of  the 
abstract  test  notations. 
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What  Information  is  Required  in  APs 
TO  Ensure  Compatible  Information 
Exchange? 

John  Aas  PEGS,  Ltd. 

Member  of  AP  206  Wireframe  Representation  Models 


I would  like  to  address  what  information  is  required 
in  APs  to  insure  compatible  information  exchange. 
Experience  has  told  us  that  definitions  which  are  too 
loose  provide  too  much  freedom  to  build  what 
implementors  like  or  what  implementors  can 
understand  from  the  document  IGES  has  a bad 
reputation  because  this  freedom  created  limitations 
that  were  not  very  good — the  information  exchanges 
between  systems  were  incompatible.  A lot  of  effort 
is  going  into  STEP  to  remedy  these  weaknesses — to 
fill  the  openings  and  to  make  it  less  likely  that 
people  will  interpret  into  the  documents  what  they 
would  like  to  do.  What  I would  like  to  present  today 
is  the  fact  that  despite  this  effort,  there  are  still  some 
openings  for  interpretation  that  need  to  be  filled.  I 
will  point  out  a few  that  the  AP  206  team  has  found 
by  experimenting  and  doing  actual  data  exchanges. 
Due  to  limited  time,  I will  address  one  aspect  within 
each  of  these  issues: 

1.  conversion 

2.  entity  mapping 

3.  application  model  characteristics  preservation. 

To  begin  with,  STEP  does  not  have  a clear  picture  of 
how  the  implementation  of  STEP  technology  should 
be  done.  Our  team  thinks  that  implementations  will 
be  done  which  produce  systems  of  a special  kind 


which  have  a well  defined  boundary  but  relatively 
small  scope,  e.g.  a manifold  solid  modular.  Users 
will  create  product  definitions  and  extract  product 
definitions  from  a system  of  the  same  specialized 
type — essentially,  systems  that  have  equivalent 
functional  capabilities.  The  data  exchanges  in 
general  will  not  be  across  functional  capabilities. 
What  does  equal  mean?  Due  to  practical  limitations, 
a user  will  actually  be  sending  models  between 
different  applications  because  the  implementation 
may  internally  represent  information  differently  than 
the  STEP  representation.  Secondly,  a systems 
implementor  wants  his  system  to  be  unique.  The 
system's  distinguishing  features  are  what  sell  the 
implementation.  In  STEP,  we  are  talking  about  the 
structure  for  representing  product  information  where 
we  are  exchanging  models  between  different  kinds  of 
systems  internally  even  though  they  have  the  same 
STEP  interface.  APs  have  to  find  a way  of 
describing  a conversion  fixjm  one  type  of  system  to 
another.  APs  need  to  address  the  conversion  aspect 
and  not  just  leave  the  interpretation  of  what  an 
acceptable  conversion  is  to  individual  users  and 
implementors  of  the  AP  documents. 

The  next  issue  is  the  mapping  of  application 
requirements  to  the  STEP  integrated  resoinces.  First 
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an  example  from  Part  205,  there  are  three  types  of 
surfaces  geometric  representation  models  supported 
in  Part  205.  The  three  representation  models  are  3-D 
surfaces  model,  a face-based  surface  model,  and 
shell-  based  surface  model.  Typically  in  a direct, 
simple  transfer,  a surface  modeler  has  the  full 
topology.  With  instructions,  we  can  make  this 
topology  into  a bounded  surface  model  and  it  can  be 
converted  to  a manifold  model — i.e.  a model  with 
no  more  than  two  surfaces  meeting  at  each  edge.  If 
you  have  surfaces  that  meet  at  more  than  two  faces  at 
the  same  edge  you  can  map  that  to  a non-manifold 
surface.  There  is  the  freedom  to  map  a geometric 
representation  model  of  one  category  into  several 
alternative  representations. 


Main  Points: 


Implementation  systems  may  continue  to  use  internal 
representations  of  product  models  that  differ  from 
STEP. 

An  AP  should  describe  the  requirements  on  any 
conversion  of  one  entity  representation  to  another 
and  not  leave  this  open  to  die  interpretation  of 
implementors  of  the  AP. 

Actual  testing  of  data  exchanges  of  the 
representations  prescribed  by  the  AP  should  be  done 
prior  to  AP  standardization. 

Need  to  accept  that  implementations  will  convert 
between  their  internal  representations  and  a STEP 
representation  and  define  the  constraints  needed  to 
ensure  meaningful  data  exchange. 


Recommendations: 


♦ Limitations  of  the  way  STEP  integrated  resources 
are  used  are  necessary. 

♦ Where  possible,  only  a single  mapping  should  be 
used. 

♦ Any  options  should  be  fully  described  including 
any  implications  on  conversions. 

♦ The  existence  of  undesirable  flexibility  is  very 
valuable  feedback  to  the  definition  of  the  AP 

and  this  should  go  into  an  AP.  The  contents  of  APs 
should  include  enough  to  plug  all  holes. 

♦ We  should  perform  manual  testing  on  the  AP. 

♦ We  need  to  make  sure  that  all  available 
knowledge  will  be  reflected  in  the  AP  documents. 

♦ We  must  collect  experience  with  AP  guidelines. 

♦ We  should  collect  and  document  all  of  our  bad 
experiences  and  share  them  with  STEP  developers 
and  users. 

Question  and  answer  period. 
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Common  Methods  for  PDES,  Inc. 


Steve  Ryan 

Core  Mechanical  Project  - PDES,  Inc. 

Mary  asked  me  to  talk  a little  about  common 
methods  meetings  that  were  held  within  PDES,  Inc. 
during  the  last  3-4  months  starting  in  December  of 
1991.  The  objective  of  this  project  was  to  bring  a 
half  dozen  project  leaders  together  to  agree  upon  the 
techniques  that  each  of  the  projects  would  use  in  the 
future.  We  looked  at  the  different  approaches  that  we 
were  using  to  develop  and  test  CDIMs  (Context 
Driven  Integrated  Models).  Within  PDES,  Inc.  we 
have  been  doing  this  to  help  us  provide  feedback  into 
the  STEP  community  on  the  problems  with  resource 
models.  However  we  also  consider  the  CDIMs 
precursors  to  APs.  The  idea  of  this  project  was  to 
look  at  the  different  approaches  we  have  been  using 
with  the  five  projects  going  on  in  PDES,  Inc.  that  are 
developing  APs  or  will  be  developing  APs.  We  want 
to  make  them  look  more  cohesive  and 
comprehensive,  five  separate  application  projects. 
Each  had  separate  ways  of  doing  the  job. 

A comparison  was  made  between  the  CDIM  and  the 
AP  processes.  We  found  considerable  similarity  at 
several  steps  in  these  processes  even  though  the 
overall  objectives  of  CDIM  & AP  processes  were 
very  different  We  determined  that  the  scope  and 
requirements  activities  did  not  need  to  differ. 

[No  further  transcript  is  available:  the  tape  was 
overwritten  inadvertently.] 
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Developing  and  Validating 

Marine  Industry  Application  Protocols 

Kent  Reed  NIST  (on  behalf  of  NIDDESC) 

Member  of  Navy  Industries  Digital  Data  Exchange  Standards  Committee 


I am  in  the  Building  and  Fire  Research  Laboratory 
at  NIST  (National  Institute  of  Standards  and 
Technology)  and  I am  here  to  report  on  work  done 
by  the  Navy  Industry  Digital  Data  Exchange 
Standards  Committee  (NIDDESC)  which  is  a 
cooperative  effort  of  Marine  Y ards  and  the  U.S. 
Navy’s  Sea  System  Command.  This  is  work  I 
participate  in  as  a technical  advisor,  most  of  the 
"hard"  work  is  done  by  others.  These  individuals 
include;  Doug  Martin  from  NASSCO,  the  model 
owner  for  a number  of  proposed  STEP  application 
protocols;  Mike  Gerardi  from  Bath  Iron  Works  and 
Mike  Polini  from  Jonathan  Industries;  Rick 
Lovedahl  from  Lovedahl  and  Associates;  and  Dan 
Wooley  from  Newport  News  Shipbuilding.  A 
number  of  the  testing  items  I will  discuss  are  actually 
being  done  by  Subhash  Ramachandran  and  Bill 
Schmidt  of  Angle,  Inc.;  Jack  Brainen  and  Lisa  Deeds 
of  David  Taylor  Research  Center;  and  others. 

The  Navy  work  is  broadly  scoped.  A series  of  six 
application  protocols  were  proposed  by  NIDDESC  in 
Oslo  as  STEP  work  projects.  The  AP  proposals 
cover  the  topics  of  ship  piping  systems;  heating, 
ventilation  and  air  conditioning  systems;  and  ship 
electrical  raceway  systems;  ship  structural  systems; 
ship  outfit  and  furnishing;  and  ship  parts  libraries. 
All  share  the  common  heritage  in  building  industry 


applications. 

All  of  these  applications  have  a design  context.  All 
focus  primarily  on  the  context  of  exchange.  The 
Naval  Sea  Systems  Command  exchanges  design  data 
with  its  yards  and  the  yards  exchange  data  with  other 
yards  as  the  products  are  designed.  All  the 
application  protocols  have  to  deal  with  configuration 
management  of  the  systems  and  parts  as  they  are 
being  designed. 

We  have  closely  coupled  elements  in  each  one  of 
these  application  protocols.  For  example,  the  scope 
and  requirements  relate  to  the  application  reference 
model  (ARM),  and  the  application  reference  model 
maps  into  the  application  interpreted  model  (AIM). 
Test  purposes,  performance  requirements,  and  other 
AP  components  are  elements  that  we  haven't  talked 
about  yet.  They  are  all  strongly  interrelated  but  in 
fact  they  are  developed  separately.  Some  of  our 
problems  are  due  to  the  fact  that  multiple  versions  of 
application  models  are  under  development  at  any  one 
time.  There  are  parallel  activities;  one  laboratory 
may  be  analyzing  one  version  of  the  model  while  its 
developer  is  working  on  the  next  version  of  the 
model  so  both  teams  may  be  looking  at  the  next  and 
preceding  versions.  At  the  same  time,  the  multiple 
application  protocols  have  common  requirements  as 
in  the  case  of  distribution  systems  for  piping  systems 
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and  for  heating,  ventilation  and  air  conditioning. 
There  is  a massive  amoimt  of  documentation  to 
manage  in  the  existing  paper-based  model 
development  world.  It  is  a massive  problem  to 
maintain  the  consistency  of  that  documentation.  Let 
us  look  at  some  of  these  elements.  Partially 
overlapping  scope  and  requirements,  developed  by  a 
part  owner,  are  reviewed  and  revised  by  NIDDESC 
members  at  many  different  meetings.  NIDDESC 
validates  this  work  by  going  to  different  domain 
experts  to  examine  the  scope  and  the  requirements. 

If  they  agree  that  these  requirements  are  really  what 
needs  to  be  specified  for  this  application,  we  proceed 
to  the  next  step  in  the  AP  development  process. 

Over  time  we  have  had  trouble  keeping  the  scopes 
and  requirements  consistent  with  the  existing  state  of 
the  models. 

Definitions  are  handled  in  much  the  same  way.  The 
part  owner  has  the  primary  responsibility  for  either 
writing  or  collecting  definitions.  Definitions  are 
revised  in  light  of  model  testing  and  simply 
reviewing  them.  Again,  the  validation  consists  of 
having  different  people  look  at  the  definitions  and 
agree  on  what  was  meant  and  that  this  is  what  was 
required.  Part  owners  maintain  the  definitions  in  a 
word  processor.  The  difficulty  is  that  part  owners 
are  defining  everything  and  they  have  other  things  to 
do.  Typically,  there  are  not  sufficient  definitions  for 
the  entities  that  exist.  Over  time  the  model  entities 
may  drift  from  their  definitions.  Unfortunately,  the 
entities  with  missing  definitions  may  sound  like 
entities  in  other  models.  Model  owners  ^sume  that 
the  definition  for  the  series  of  APs  is  the  same.  We 
are  have  the  same  trouble  keeping  the  definitions 
synchronized,  just  as  we  have  in  keeping  the 


definitions  synchronized  within  the  Parts  of  STEP. 
We  have  trouble  just  accessing  the  definitions  that 
are  current. 

Application  reference  models  have  consumed  the 
bulk  of  the  development  effort.  Typically  the  part 
owner  developed  the  draft  ^plication  reference 
model.  These  models  have  been  built  in  NIAM, 
Nijssen's  Information  Analysis  Methodology. 

Nijssen  has  refined  his  ideas  over  10  years.  The 
software  tools  we  use  to  support  NIAM  differ  from 
each  other.  The  models  have  been  critiqued  at  many 
woikshops  and  are  now  being  revised  in  light  of  the 
testing  results.  We  have  tools  that  analyze  NIAM 
models  for  consistent  and  correct  syntax.  The  tools 
used  include  PC-IAST,  a Control  Data  Corporation 
product,  and  Ridl*  from  Intellibase  Corporation, 
Belgium.  Both  of  those  tools  generate  SQL  schemas 
which  we  use  for  model  validation  testing.  The 
ARM  graphic  representations  are  maintained  in  the 
part  owner's  word  processor  format  as  drawings. 
Their  controlling  representation  is  maintained  in  the 
repository  of  the  NIAM  tools. 

One  obvious  problem  is  maintaining  a consistent 
system  of  ARMs  across  the  scope  of  multiple  APs. 
Since  no  tool  does  everything  we  need,  we  must 
maintain  multiple  representations,  one  for  each  tool 
we  use.  To  further  complicate  matters,  the  tools 
have  different  naming  conventions.  Since  we  are 
testing  the  ARM  in  a relational  database 
environment,  we  have  no  direct  mechanism  to  test 
the  constraints.  The  major  problem  for  us  is  that  it 
takes  a tremendous  amount  of  human  interaction  to 
revise  a model  and  then  transform  these  revisions 
into  the  formats  that  each  of  the  tools  require. 
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To  move  on  to  the  ARM  to  AIM  mapping  table,  each 
one  has  been  developed  primarily  by  the  part  owner 
manually,  without  any  software  to  generate  it.  The 
mapping  table  is  then  revised  in  light  of  the  model 
enhancements  and  the  testing  results.  The  AP 
Guidelines  are  not  very  clear  on  the  requirements  for 
the  mapping  table  and  each  example  we  have  looked 
at  from  the  emerging  APs  is  different  from  the  last. 

Validation  of  the  AIM  has  proceeded  only  to  the 
extent  that  I have  tried  to  compile  some  of  the 
Express  schema  into  the  Data  Probe,  a prototype 
software  tool  which  is  available  through  the 
Manufacturing  Engineering  Laboratory  and  National 
PDES  Testbed.  Next,  we  will  populate  these  models 
with  data  from  the  ship  yards.  We  decided  that 
going  back  to  a relational  data  base  environment  at 
this  point  is  too  much  effort  The  AIM  is  maintained 
in  a word  processor.  At  some  point  we  will  need  to 
include  EXPRESS-G  drawings,  if  we  can  find  a tool 
to  build  the  EXPRESS-G  with.  I would  like  to  point 
out  that  test  case  data  used  in  AIM  testing  are  the 
same  data  that  we  used  for  ARM  testing. 

In  the  sections  on  conformance  requirements  and  test 
purposes,  I will  ONLY  speak  on  usage  test  purposes 
at  this  point  and  not  the  more  basic  structural  test 
purposes.  They  have  been  drafted  by  part  owners 
and  reviewed  and  revised  by  other  team  members. 
The  validation  of  conformance  requirements  and  test 
purposes  consists  of  a critique  by  domain  experts. 
During  this  process,  we  start  to  gather  test  case  data 
from  domain  experts  along  with  data  from  the 
testing  teams  who  use  existing  Navy  projects  as  their 
source  for  data.  For  example,  the  piping  system 
testing  information  is  coming  from  the  design  of  the 


combat  information  centerfor  the  DDG51.  In  the 
test  purposes,  their  definition  is  a problem.  The 
source  of  the  problem  is  with  the  process,  the 
requirements  have  changed  at  an  alarming  rate. 

The  primary  problem  for  us  is  lack  of  a good 
environment  to  help  us  control  of  our  information. 
We  have  multiple  representations  of  the  same  ideas 
because  we  operate  in  different  environments.  We 
have  tried  to  use  the  best  tools  but  these  offer  only 
fragmented  solutions.  We  lack  tools  that  support 
model  validation  testing.  The  tools  that  do  exist  are 
not  yet  industrial  strength;  I have  broken  them  all. 
This  is  a common  complaint  about  the  software 
industry  but  it  hurts  the  development  of  STEP  where 
so  much  complex  information  has  to  be  captured  and 
manipulated.  The  tools  are  not  validated  to  work 
correctly  so  there  is  always  the  question  "Are  we 
testing  the  tool  or  the  model?".  In  using  the 
available  EXPRESS  compilers,  a lot  of  the  problems 
have  been  worked  out  They  are  still  one-way 
processors.  We  developed  filters  to  move  from  one 
model  environment  to  another  model  environment 
but  this  is  a one-way  process.  If  we  change 
something  between  ARM  and  the  AIM,  it  is  very 
difficult  to  propagate  that  change  backwards  to  go 
back  to  earlier  steps  in  the  AP  development  process 
and  revalidate  the  changes  is  very  time  consuming. 

In  spite  of  all  this,  the  message  I would  like  to 
convey  here  is  that  we  believe  in  STEP.  There  really 
is  no  other  alternative.  NIDDESC  and  others  are 
trying  very  hard  to  make  STEP  work.  We  must, 
however,  find  ways  to  make  this  work  easier  and  less 
human  resource  intensive.  STEP  developers  need  to 
take  advantage  of  everything  that  is  available  but  I 
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don't  think  this  is  sufficient.  We  will  need  a lot  of 
automated  tools  to  get  the  job  done.  Paper-based 
management  of  AP  development  is  not  enough  and  it 
requires  too  many  scarce  human  resources.  There 
needs  to  be  a concentrated  effort  to  develop  a 
software  environment  to  support  AP  development 
This  would  be  cost  effective  and  improve  the  chances 
of  the  success  of  STEP.  The  current  documentation 
in  the  AP  guidelines  and  other  STEP  guidelines  are 
not  sufficient  to  provide  adequate  direction  to  STEP 
developers.  This  causes  each  development  project  to 
hash  through  the  same  set  of  problems  and 
ambiguity.  We  should  take  the  time  to  document  the 
best  known  practices  in  these  documents  now. 


Question  and  answer  period. 


Main  Points: 


NIDDESC  is  developing  six  highly  integrated  APs 
for  the  marine  industry. 

The  environment  for  developing  an  AP  is 
fragmented  which  makes  it  difficult  to  maintain  a 
consistent  set  of  AP  components. 

Scope,  requirements,  and  definition  are  validated  by 
reviews  from  domain  experts. 

Both  the  ARM  and  AIM  are  validated  by  populating 
the  models  with  real  world  data. 

Software  tools  support  for  AP  development  must  be 
improved. 

STEP  Guidelines  documents  must  be  enhanced  to 
reflect  lessons  learned  and  the  current  best  practices 
for  development 
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The  Roles  of  Mapping  Tables  and 
Conformance  Test  Purposes  in  STEP 
Application  Protocols 

Julian  Fowler  CADDETC Member  of  AP  201  Explicit  Draughting 

Presented  by  Jon  Owen  CADDETC  Convener,  WG6,  Conformance  Testing 


I will  go  through  the  thoughts  that  CADDETC  has 
had  on  the  role  of  mapping  tables  and  conformance 
test  purposes  in  STEP  application  protocols.  This 
has  been  largely  inspired  by  Julian  Fowler  from  his 
role  in  Part  201,  Explicit  Draughting.  First  I will 
provide  backgrotmd  and  ask  the  question  "What  is 
an  AP?"  This  answer  to  the  role  of  the  mapping 
table  and  test  purposes  is  not  guaranteed  but  it's  a 
start  We  will  examine  the  relationships  between  the 
components  of  an  AP.  Then  we  will  look  at  the  role 
of  the  mapping  table;  what  is  in  it  and  what  the 
prospects  are  for  moving  forward  on  an  improved 
representation  for  it  The  last  half  of  the 
presentation  will  look  at  the  test  purposes;  what  they 
are  and  how  they  are  documented.  Then  I will  try  to 
bring  the  importance  of  these  components  together 
in  the  summary. 

Mapping  tables  and  test  purposes  were  required 
components  of  APs  from  the  start  In  practice,  the 
work  on  APs  to  date  has  concentrated  on  the  activity 
models,  the  tq)plication  reference  models,  and  in 
particular  the  AIM  development  The  approaches  to 
the  mapping  tables  and  the  test  purposes  were  not 
agreed  to  or  documented.  That  is  evident  from  the 


differences  in  the  appearance  of  Part  201  and  Part 
203.  Julian  is  not  here  today  but  consider  all  the 
mapping  tables  he  merged  in  proposing  an  improved 
mapping  table  format  that  could  support  the 
generation  of  test  purposes  and  an  abstract  test  suite. 
I believe  the  mapping  table  format  can  and  I believe 
it  has  emerged  out  of  the  integration  processes.  How 
can  we  ensure  that  the  AIM  actually  satisfies 
everything  that  is  in  the  ARM.  I will  siunmarize 
where  we  are  and  give  a few  suggestions  on  how  to 
move  forward. 

What  is  a STEP  Application  Protocol?  It  is  a 
standard  and  a specification  of  the  data  constructs 
that  are  required  to  exchange  all  the  infonnation 
needed  to  meet  documented  industrial  requirements. 

In  practice,  an  AP  may  have  several  levels  of 
conformity.  It  is  important  that  each  clearly 
identified  need  should  be  documented  in  separate  AP 
level.  The  levels  of  functional  capability  within  an 
AP  are  kind  of  a miniature  AP  within  the  AP. 

The  STEP  implementation  combines  an  AP  and 
implementation  form,  such  as  Part  21  - the  Physical 
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File,  or  SDAI. 

An  AP  is  not  a subset  of  the  integrated  resources  as 
has  been  thought  of  in  the  past.  What  we  need  to  do 
is  to  integrate  the  AP  so  that  we  can  enable 
interoperability  between  different  APs. 

We  have  a mapping  table  that  brings  together  the 
information  requirements  in  the  ARM  and  the  AIM 
as  well  as  the  conformance  test  purposes.  The  test 
purposes  exercise  the  structures  and  requirements  in 
the  mapping  table. 

A refinement  of  the  previous  slide  is  that  we  can  see 
an  AIC  as  a complete  testable  element  which  can  be 
shared  by  more  than  one  AP. 

The  idea  behind  the  mapping  table  is  to  document 
the  correspondence  between  what  was  in  the 
information  requirements  in  the  AIM.  The  mapping 
table  is  really  providing  a link  between  an 
application  domain  and  its  language.  An 
observation  is  that  the  length  and  the  level  of  detail 
in  the  mapping  tables  will  depend  on  the  details  of 
the  AP  information  needs.  If  you  have  a very 
detailed  Application  Reference  Model  then  false 
detail  will  be  presented  in  the  mapping  table.  Part 
201  has  a two-level  mapping  table.  The  content  and 
layout  has  not  been  agreed  to  but  this  is  one  way  to 
produce  a mapping  table.  Each  construct  of  the 
ARM  is  matched  to  a primary  construct  in  the  AIM. 
That  worics  for  Part  201  and  it  may  be  sufficient  for 
an  AP  with  a mapping  that  has  a direct 
interpretation  of  integrated  resources.  APs  such  as 
Part  204,  Part  205,  and  Part  206  have  requiranents 
which  are  very  close  to  the  representation  provided 


by  Part  42. 

The  mapping  table  to  date  has  tended  to  be  a - I've 
got  something  here  in  the  ARM  that  I need  to  satisfy 
and  here  is  some  stuff  either  of  the  one-liner  or  a 
collection  of  things  that  fulfill  it.  It  might  be  quite 
useful  in  our  environment  to  be  able  to  have  the 
mapping  properly  directed  back  to  the  ARM.  If  I 
have  an  instance  of  something  in  my  physical  file, 
what  is  that  actually  doing?  You  get  a pointer  back 
that  says  something  like  in  application-speak  "1  am 
satisfying  this  requirement". 

We  have  the  AIM  in  the  EXPRESS  language.  It  is 
useful  to  make  the  mapping  tables  computer 
processable.  The  question  here  is  do  we  actually 
need  a standardized  formal  method  for  documenting 
requirements?  Possibly..Probably... 

Conformance  test  purposes  are  the  starting  point  for 
denoting  an  abstract  test  case.  It  provides  the 
objective  that  you  are  trying  to  test  If  you  take  the 
test  purpose,  a particular  bit  of  the  mapping  table 
exercised  by  the  test  purpose,  and  then  the 
information  requirements  from  the  AIM  that  it 
naturally  brings  in,  that  specifies  the  things  that  you 
need  in  order  to  be  able  to  test  a particular  test 
purpose.  Conformance  test  purposes  identify  all 
discrete  options  that  are  in  the  applicatioiL 

That  process  for  defining  conformance  test  purposes 
is  documented  in  WG6  document  N26  written  by 
Julian  Fowler  for  this  workshop.  It  is  feasible  to 
produce  the  whole  set  of  required  test  purposes  from 
the  AIM  automatically.  We  have  rules  for  mapping 
EXPRESS  to  generate  the  stand-alone  discrete 
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options  to  be  tested.  The  other  half  has  yet  to  be 
documented.  Just  because  of  this  problem  with  the 
mapping  table — we  can  express  test  purposes  in 
STEP-speak  but  not  as  it  relates  back  to  the  ARM  in 
application-speak.  A word  on  the  test  purposes  for 
AP  201.  Test  purposes  were  grouped  in  a particular 
way.  Test  groups  are  structured  by  units  of 
functionality,  then  there  is  a series  of  test  purposes 
for  product  associations.  This  is  not  AP  201 
specific.  We  anticipate  that  every  AP  is  going  to 
have  these  test  groups  because  they  are  part  of  Part 
41. 

Next,  how  are  the  test  purposes  being  documented? 
Test  purpose  name  is  under  the  corresponding  AIM 
entity.  Where  that  entity  participates  in  a role  is  the 
context  tmd  they  are  listed.  There  is  a close 
relationship  between  the  mapping  table  and  the  test 
purposes.  CADDETC  has  an  algorithm  for 
generating  test  purposes  from  a given  EXPRESS 
specification.  It  does  not  include  consideration  of 
defects  in  global  and  local  rules  so  that  if  there  is  an 
attribute  that  says  optional  the  program  will 
generate  a "present"  or  "not  present"  with  a local 
rule  that  says  where  it  takes  it  - which  means  it's 
always  there.  But  in  practice  the  algorithm 
generates  the  maximum  set. 

The  mapping  tables  include  better  identific^on  of 
the  AIM  to  ARM  mapping  rather  than  the  ARM  to 
AIM.  The  technique  and  a given  mapping  table  can 
fulfill  that  because  we  have  this  half  of  the 
automation.  If  we  can  get  the  other  half,  then  we 
can  move  die  conformance  test  purposes  in  the  AP 
into  the  abstract  test  cases  because  there  is  a formal 
and  sufficient  basis  for  doing  diat.  The  technique 


will  always  be  reproducible. 

Fmally,  a brief  word  on  what  has  been  called  usage 
test  purposes.  There  are  tests  around  which 
exercise  specific  combinations  and  potentially 
complex  combinations  of  data  in  the  AP.  We  believe 
that  such  complex  test  purposes  should  be  used 
before  you  move  into  conformance  testing.  For 
example,  if  I am  a customer  who  uses  this  AP  to  do 
the  sort  of  things  that  I want,  then  I want  a system 
that  does  A3,  & C.  You  can  specify  a usage  test  to 
make  sure  that  you  get  the  information  you  need 
within  a particular  AP  or  a particular 
implementation.  You  can  then  say  this:  Is  it  what  I 
need?  Does  the  model  actually  do  it  correctly?  So 
the  recommendation  here  is  that  each  of  these  test 
purposes  should  be  used  as  part  of  the  AP  evaluation. 

Question  and  answer  period. 

Main  Points: 

An  AP  may  have  several  levels  of  conformity;  each 
should  be  documented  in  separate  AP  level. 

Mapping  tables  bring  together  the  information 
requirements  in  the  ARM  and  the  AIM  solutions  to 
them  as  well  as  the  conformance  test  purposes. 

Conformance  test  purposes  are  the  starting  point  for 
specifying  abstract  test  cases. 

Mapping  tables  include  better  identification  of  the 
AIM  to  ARM  mapping  rather  than  the  ARM  to 
AIM. 

We  can  move  the  conformance  test  purposes  in  the 
AP  into  the  abstract  test  cases  because  thwe  is  a 
formal  and  sufficient  basis  for  doing  that. 
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Summary  & Workshop  Recommendations 


Working  Sessions 


The  pMticipants  were  divided  into  three  working 
sessions: 

L Ensuring  Requirements  Traceability 

Steve  Brett,  Mitchell  Gilbert , Jochen 
Haenisch,  Mary  Mitchell,  Kurudi 
Muralidhar  Murali,  Kent  Reed 

n.  AP  Validation  Requirements  and 
Documentation 

Diane  Allen,  Allison  Barnard,  Bill  Burkett, 
Larry  McKee,  Haidee  Rapacke,  Bob  Street, 
Arme  Williams 

HL  AP  Project  Management  and 
Planning 

Jon  Aas,  John  Barnes,  Peter  Kruse,  Mark 
Palmer,  Sandy  Ressler,  Bill  Russell 


Action  Items 


The  action  items  which  were  generated  during  the 
workshop  fell  into  four  categories: 

♦ improvements  to  existing  documents,  such  as  the 
AP  Guidelines  and  AP  QualiBcation  Manual 

♦ refinements  to  the  AP  development  process, 

♦ requirements  for  additional  WG4  documents  or 
actions  needing  WG4  attention,  and 

♦ actions  which  need  attention  by  groups  outside  of 
WG4. 


Identified  needs  and  potential 
sources  to  assume  the  responsibility: 

♦ Quality  management  structure  which  covers  the 
AP  development  process  and  STEP  methods. 
(WG4^1  project  lead  and  IPO  Dictionary 
methodology  committee  coordination) 

♦ Identify  the  rights  and  responsibilities  for  an  AP 
planning  project.  (PMAG  AP  Coordinator) 

♦ Template  work  break  down  structure  for  AP 
project  planning.  This  should  identify  expectations 
on  time,  manpower  resources,  coordination 
requirements  and  necessary  approvals.  Sources  for 
initial  material  include  M.  Mitchell's  Development 
Plan  for  Application  Protocols  Mechanical  Parts 
Production  and  S.  Ryan's  APDBE  work  breakdown 
structure.  (PMAG  AP  Coordinator) 

♦ WG4  projects,  resource  model  projects  and  AP 
projects  must  maintain  an  accurate  schedule  which 
can  be  distributed  by  SC4.  (WG4  Convenor,  & 
project  leaders) 

♦ WG4  should  document  how  its  priorities  are 
established.  (WG4  project  leaders) 

♦ Additional  criteria  to  establish  WG4  work 
priority:  this  should  include  the  identification  of  AP 
capabilities  needed  by  multiple  industries.  (PMAG 
AP  Coordinator) 

♦ Rules  and  procedures  to  defined  on  how  gaps  in 
the  integrated  resources  are  to  be  filled.  TheAP 
guidelines  assume  integrated  resources  are  sufficient 
to  satisfy  the  application  requirements.  A principal 
criteria  for  selecting  the  initial  release  APs  was  the 
existence  of  draft  resource  models.  ( i.e.  there  is  no 
guidance  on  how  new  integrated  resource  models  are 
initiated  and  added  to  a release  of  STEP)(WG4, 
PMAG) 

♦ Rules  and  procedures  to  guide  the  development  of 
AIC^.  How  specific  should  AICs  be?  How  to  modify 
an  existing  AIC  (P6,  AP  Integration  Project) 
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♦ Establish  criteria  is  used  to  determine  the 
commonality  of  requirements  and  the  utility  of 
existing  AICs  and  APs  for  use  by  additional  APs  (AP 
integration) 


Improvements  to  existing  documents 
identified: 

♦ Identify  the  minimum  set  of  information  needed 
to  evaluate  the  scope  of  an  AP.  This  is  needed  for 
the  AP  project  approval  process.  (AP  (Salification 
Manual  & AP  Guidelines  documents : Mark  Palmer 
with  submissions  by  WG4/P1). 

a)  CD  comments  by  member  countries; 

b)  for  AP  project  approval. 

♦ Specify  the  AP  components  that  are  part  of  the 
submission  for  CD  Comment  ballot  by  PMAG  and 
SC4.  (AP  Guidelines) 

♦ Define  the  required  contents  of  an  ARM 
population  section  of  the  AP  validation  report  (AP 
(Salification  Manual  and  AP  Guidelines) 

♦ Define  what  constitutes  an  adequate  ARM  and 
AIM  population  coverage.  (AP  (Salification  Manual 
and  AP  Guidelines) 

♦ Document  the  IR  version  references  used  in  the 
AIM.  An  AP  should  list  the  version  (N-number)  of 
resource  parts  in  mapping  table  and  references  in  AP 
bibliography.  (AP  Guidelines) 

♦ Define  the  required  contents  of  the  AIM 
compilation  report  (AP  Guidelines) 

♦ Specify  what  EXPRESS  compilers  are  approved 
for  use  in  checking  the  correcmess  of  the  EXPRESS 
in  the  AIM  (AP  Guidelines  & AP  Qualification 
Manual  with  guidance  from  WG6) 

♦ Define  how  small  of  a market  sector  can  AP 
target  An  example  is  Navy  shipbuilding;  is  this  a 
large  enough  sector  to  warrant  international 
standardization  by  ISO  TC184/SC4  (AP  Guidelines) 

♦ Provide  guidance  on  conformance  requirements 
and  test  purposes  to  AP  developos  with  respect  to 
conversion  between  shape  (geometric) 
representations.  APs  need  to  answer  questions  like: 
when  is  conversion  is  allowed;  what  test  purposes 


are  needed  are  needed  to  assess  the  result  of  a 
conversion,  and  what  are  the  acceptance  criteria. 
(AP  Guidelines,  Guidelines  for  the  Development  of 
Test  Purpose ) 

♦ Define  the  required  contents  of  an  AIM 
population  section  of  the  AP  validation  report . (AP 
(Salification  Manual  and  AP  Guidelines) 

♦ Define  boiler  plate  text  for  integrated  resource 
interpretation  report  (Supplemental  Directives) 

♦ Specify  an  appropriate  introduction  to  the  ARM- 
AIM  mapping  table.  Should  material  from  the 
integrated  resource  interpretation  report  be  used  as 
the  basis  for  the  introduction  to  the  ARM-AIM 
mapping  table?  (Supplemental  Directives) 

♦ Direct  AP  developers  to  document  the  intent  of 
selected  resource  constructs  in  the  integrated 
resource  interpretation  report  (AP  Guidelines) 

♦ State  that  the  AAM  to  ARM  mapping  report 
needs  to  be  part  of  validation  report  (AP 
(S^dification  Manual  and  AP  Guidelines) 

♦ Require  AP  team  to  describe  their  information 
modeling  expertise  (AP  (Salification  Manual) 

♦ Require  proof  that  a structured  approach  to 
information  gathering  was  used  in  developing  the 
AP  requirements.  (AP  Qualification  Manual  and  AP 
Guidelines) 

♦ Require  proof  that  a method  that  ensures 
traceability  of  requirements  fixrm  scope  throughout 
AP  development  process  was  used.  An  AP  shotild 
demonstrate  that  this  has  occurred;  QFD  is  one 
possible  technique.  (AP  Guidelines) 

♦ Update  the  Supplement  Directives  to  reflect  the 
current  AP  development  process.  (Supplemental 
Directives) 

♦ State  how  to  assess  the  correspondance  of  AIM 
mapping  table  to  test  purposes.  (AP  (Salification 
Manual) 

♦ Provide  guidance  on  how  to  assess  validity  of 
requirements.  In  information  gathering  stage, 
domain  experts  may  report  plarmed  not  actual 
requirements.  (AP  Qualification  Manual) 

♦ State  that  to-be  information  requirements  will  not 
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be  allowed  if  the  transformation  from  the  as-is  to 
the  to-be  cannot  be  produced.  (AP  Qualification 
Manual  and  AP  Guidelines) 

♦ Define  how  an  AP  is  to  demonstrate  the 
correspondence  between  the  scope,  units  of 
functionality,  and  requirements.  The  AP  scope 
states  the  overall  required  functionality  of  the  AP 
and  Clause  4 states  the  information  requirements. 
(AP  Qualifications  Manual) 

♦ Identify  what  criteria  is  used  to  determine  if  an 
AP  scope  unique.  (AP  Guidelines) 


Required  action  external  to  WG4: 

♦ Develop  consensus  on  criteria  for  committing 
resources  to  subsequent  work  which  extends  the 
functionality  of  STEP.  (SPAG,  IPO  Steering 
Committee;  National  TAG) 

♦ Develop  acceptance  criteria  for  implementations 
of  EXPRKS  compilers.  (WG6) 

♦ Provide  a list  of  approved  and  acceptance  tested 
EXPRESS  compilers  to  all  STEP  Part  developers. 
(WG5  Express  project) 

♦ Decide  on  the  inclusion  of  assertions  in  generating 
conformance  test  purposes.  (WG6) 

♦ Develop  procedure  for  review  and  approve  AP  test 
purposes.  (WG6) 

♦ Define  a procedure  for  validating  EXPRESS 
compilers.  (WG6) 

Mark  Palmer  presented  some  of  these 
recommendations  at  the  Wednesday  PMAG. 

AP  Qualification  Manual  additional 
criteria  proposed 


For  validation  of  the  AP  requirements  clause: 

♦ Proof  that  there  was  a trained  and  experienced 
information  requirements  staff. 


♦ Proof  that  the  project  employed  a structured 
information  gathering  technique.  (The  AP  project 
should  submit  questionnaire  or  other  documented 
method  for  group  1 review.) 

♦ Demonstrate  the  AP  can  support  a broad  industrial 
^plication  by  obtaining  actual  data  from  multiple 
enterprises  and  more  than  one  country  and 
populating  both  the  ARM  and  AIM  with  it. 

♦ Document  reviews  of  the  capabilities  within 
existing  commercial  systems  which  support  all  or 
part  of  the  application  scope. 

♦ Proof  of  the  existence  of  a plan  that  was  used  for 
coordinating  the  requirements  analysis. 

Are  the  requirements  within  the  AP  scope? 

♦ Document  with  the  external  reviewer  evaluation, 
the  use  a structured  method. 

♦ Qieck  that  the  AP  project  has  established  priority 
within  the  AP  framework. 

Further  Discussion  Required: 

If  an  AP  goes  beyond  existing  practices  and 
provides  a to-be  process  model,  how  should  this  be 
validated? 

Should  the  ARM-AIM  mapping  require  that 
transitive  closure  be  proved? 

Should  the  mapping  structure  provide  the  exactness 
that  would  allow  the  ARM  to  be  reverse  engineered 
from  the  AIM  as  a verification  technique? 


Traceability  of  Requirements 


♦ The  lack  of  traceability  can  be  tied  to  the  need  to 
improve  the  guidance  on  how  an  AP  is  to  document 
that  the  AP  development  process  was  followed. 

♦ Some  flexibility  should  be  allowed  in  how  to 
demonstrate  traceability.  Equivalent  approaches  are 
likely.  We  lack  experience  to  select  any  specific  one 
as  the  best  method  to  use.  Especially  for  the 
normative  elements  ofanAP,  we  need  to  establish 
what  the  minimum  requirement  is. 

♦ All  requirements  should  be  traceable  to  an  ARM 
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object.  If  it  cannot  be  traced  then  it  is  probable  that 
the  resulting  AP  cannot  satisfy  this  requirement. 

♦ Develop  usages  tests  based  on  expert  interviews 
and  existing  application  system  analysis  and  expand 
the  constraints  uncovered  by  these  into  AIM 
reference  path  queries.  These  frequently  reflect 
important  aspects  which  are  necessary  to  satisfy 
application  requirements. 

♦ Extend  the  current  mapping  table  structure  to 
eliminate  the  following  limitations 

a)  the  current  structure  is  unable  to  express 
relationships  including  cardinality, 

b)  ARM  many-to-many  relationships  are  not 
cleanly  represented  in  the  mapping  table,  and 

c)  constraints  in  the  AIM  which  have  significance 
in  the  ARM  are  not  represented 
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REQUIREMENTS  FOR  AP  VALIDATION  & THE  AP  VALIDATION  REPORT 
Larry  McKee 


Our  group  discussed  the  following  topics: 
documenting  the  application  interpretation;  the 
extent  that  an  AIM  should  be  test  populated  with 
data;  assessment  of  conformance  requirements  and 
test  purposes.  An  AIM  should  be  test  populated  with 
data  and  that  population  should  be  provided  in  a 
report  that  at  a minimum  should  access  the  degree  of 
coverage  of  the  population,  meaning  how  much  of 
the  AIM  did  you  actually  populate.  The  minimum 
data  used  in  the  population  should  be  the  same  part 
used  to  populate  the  ARM.  Another  piece  of 
documentation  for  the  AIM  is  a list  of  the  numbers 
of  the  integrated  resources  Parts  that  were  used  to 
make  the  AIM.  The  AP  document  should  state 
exactly  what  versions  of  the  Parts  were  used  in 
building  the  AIM  so  that  there  is  no  doubt  of  the 
version  used  in  the  AIM. 

Another  requirement  is  a compile  of  the  EXPRESS 
for  the  AIM.  The  results  of  that  compile  are  to  be 
provided  in  documentation.  For  the  group  3 
evaluation  of  conformance  requirements  and  test 
purposes,  especially  for  test  purposes  it  doesn't  make 
sense  to  evaluatitm  them  by  themselves.  They  need 
to  be  evaluated  against  the  AIM  to  make  sure  all  the 
required  test  purposes  are  there.  For  conformance 
requirements,  the  test  purpose  should  indicate  which 
of  the  AFs  various  implem^tadon  levels  they  apply 
to. 


One  of  the  validations  we  talked  about,  but  we  did 
not  discuss  how  to  document,  was  determining  what 
the  conformance  requirements  for  the 
implementation  level  are  and  that  it  should  map  into 
a closed  set  of  entities.  When  you  conform  to  this 
implementation  level,  it  uses  a fixed  and  closed  set 
of  entities  that  you  must  conform  to. 

Documenting  the  Validity  of  an  AP 
One  form  of  documentation  should  be  the  results  of 
the  ballot  of  the  scope,  AAM  and  ARM  along  with 
comment  resolutions.  Another  part  of  the  report 
should  be  the  ARM  population  report  with  a 
coverage  analysis  which  identifies  how  much  of  the 
model  was  populated.  Furthermore,  it  describes 
why  things  were  not  populated.  The  report  should 
list  the  N-numbered  integrated  resource  parts  used  in 
the  AIM  compilation  report  The  AIM  population 
report  should  be  similar  to  the  ARM  population 
report;  it  should  show  that  at  least  the  ARM  test  part 
was  used  in  the  population  and  the  results  of  the 
population  should  be  described.  Test  purpose  to 
AIM  construct  mapping  report  shows  that  for  every 
enumerated  construct  in  the  AIM  you  have  a test 
purpose  that  exercises  all  optional  and  enumerated 
values.  The  integrated  resource  interpretation  report 
describes  how  the  interpretation  was  done  and  where 
the  mapped  elements  came  from.  There  was  a 
proposal  for  WG6  to  perform  test  purpose  review 
and  that  the  results  of  that  review  from  WG6  the  AP 
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developers  will  be  responsible  for  covering  any  responses  because  the  P member  countries  must 

comments  that  WG6  had  against  the  test  purposes  reply, 

and  how  they  were  resolved.  On  the  AAM  to  ARM 

mapping  report,  assuming  the  use  of  IDEF  0 for  Question  and  answer  period. 

your  AAM,  each  one  of  the  inputs  and  outputs 

should  be  mapped  into  ARM  data  constructs.  It 

won’t  be  one-to-one  mapping  but  it  should  show  that 

the  in-scope  inputs  and  outputs  are  covered. 

Another  part  of  this  document  essentially  takes  what 
has  been  called  complex  test  purposes  and  shows 
how  they  are  used  against  the  AIM  to  produce  the 
data  which  satisfies  the  input  and  output 
requirements.  Those  are  the  things  we  have 
identified  in  the  model  validation  methodology.  The 
fundamental  thing  we  need  to  agree  on  with  respect 
to  AP  validation  is  to  get  an  up-front  validation  of 
the  scoping  requirements  to  eliminate  the  problems 
we  have  seen  in  all  of  the  APs  up  to  now  with 
requirements  extensions  and  contractions.  The  AP 
projects  must  make  sure  they  have  an  agreed  upon 
set  of  requirements. 

The  other  thing  discussed  was  where  the  ballot 
comments  came  from.  Industry  expert  reviews  tend 
to  be  done  within  the  confines  of  your  group,  within 
U.S.,  European,  or  Asian,  etc.;  they  tend  to  not 
extend  into  the  entire  ISO  arena.  Using  the  ballot 
mechanism  would  force  an  international  review  of 
scope  and  requirements.  This  ballot  would  not  be  an 
approve  or  rejection;  the  intent  is  to  just  verify  that 
none  of  the  requirements  have  been  left  out  The 
problem  we  have  experienced,  is  ttiat  if  you  send  a 
document  out  to  all  the  P member  countries  for 
comment  and  not  for  ballot,  we  do  not  get  any 
response.  If  we  send  the  AP  scope  and  requirements 
out  for  ballot,  we  will  probably  get  scHne  meaningful 
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AP  Development  and  Validation  Issues 


1.  Verification  and  international  consensus  of  AP  scope  and  requirennents 

- process  and  procedures  defined 

2.  ARM  development  and  validation 

- specificity  of  industry  semantics  and  detail  (granularity)  of  ARM 

- rules  for  defining  "units  of  functionality"  (UoFs) 

- verification  of  UoFs 

- role  of  "functional  building  blocks"  to  extend  core  ARM 

3.  AIM  development  and  validation 

- ensuring  traceability  from  the  AP  requirements  to  AIM  to  ATS 

- rules  and  tools  for  preparing  the  AIM  long  form  from  short  form 

- how  to  assess  the  AIM  short  form,  how  to  assess  the  AIM  long  form 

- procedures  for  building  and  validating  reusable  AlCs 

4.  Development  and  validation  of  test  purposes,  abstract  test  suite  (ATS),  and 
executable  test  cases 

- procedures  for  deriving  test  groups  and  test  purposes 

- required  format  and  granularity  of  conformance  test  purposes 

- consensus  on  the  value  and  role  of  basic  test  purposes  and  complex 
test  purposes 

- how  to  assess  the  conformance  requirements  and  test  purposes 

- viability  of  using  the  same  test  purposes  and  conformance 
requirements  to  generate  an  ATS  for  an  exchange  file  implementation 
and  an  ATS  for  a distributed  databases  implementation 

• rules  and  procedures  for  producing  an  ATS 

5.  AP  development  and  documentation 

- expanded  mapping  table:  understandability  and  simplify  the 
enumeration  of  test  purposes 

- enforcing  validation  of  APs  and  documenting  the  results 

- inclusion  of  standardized  queries  for  obtaining  the  ARM  concepts  from 
the  AIM 

- contents  and  role  of  AP  usage  guide 


Mirk  Ptimar/NIST 
Apri  13. 1902 


0^ 

o 

H 

C/) 

U 

s 

H 

O 

u 

HH 

o 

> 

53 

H 

O 

Z 

HH 

O 

cu 

Q 


Kurudi  Muralidhar  Murali  ITI  -Ann  Arbor 

Member  of  WG6,  Conformance  Testing 


EIP  Needs  and  Requirements 


IumimArjwj 

Cl 

— rr 

<3 

s 

[T 

i-i 

yi 

uiatate ‘sucqBOtddB  pyxM«iQ 

□c 

1 

s 

y 

i 1 

luiQiiBid  *ddv 

3 

o 

3 

i 

1 

S 

i 

sumsAs  ‘suoQBOfidde  jeinpo^ 

0 

S3 

s 

S 

5 

|0  Xifuienb  8QJ^  eipuBH 

s 

1 

t 

1 

vpp  lo  ssaooenufBU)  uLNi  Ouoi 

r 

r 

L_ 

□ 



t 

«>tnpotu/e0pei^BOu^  10  OBrmu 

o 

m 

o 

't 

1 

spjepuejs  eOupqajeiiJi  wmo 

o 

i 

1 

• 

o 

s 

i 

1 

1 SpjepUBlS  SUOI|B3|UnUAUBO 

b 

• 

<1 

1^ 

•• 

S 

T1 

SSNVJJKMH 

• 

tAi 

A 

fV 

i 

1 

lx  1 

1 

IHI 

f 

lU 

lA. 

j 

1 

_ 

O 

& 

O 

1 

? 

1 

w 

O 

w 

(0 

u. 

f 

o 

0 

8 

1 
cc 

ABSOLUTE  tMPOITANOi 

Interaction  Points  from  Roof 

1 

1 

(T 

V. 


Continuous  Quality  Improvement 


EIP  Needs  and  Requirements 


luauMOMRij  epQ 

• 

□ 

r— T— 

f<I 

< 

H 

IE 

uiOsAs  ‘suoiieoidde  frnmqmQ 

T 

*jadoje;ij|Atapu!  iiiic^pud  'ddy 

i 

i 

5 

1 

siuaisXs  *stinipij|ckli 

Ob 

s 

• 

epBp  «o  AnuBie  aA^ 

e 

3 

Z 

Enp  >0  ssaaoB/iiipui  uuet  Ouoi 

J 

t 

t 

98|npoiivaflp«|Mou)|  ^0  esnevj 

o 

< 

< 

0 

b 

5 

n 

spJBpunS  aGuBgoieiui  e^eo 

0 

< 

e 

0 

5 

T 

1 

1 spiepueis  suo.aeofunujujoo 

!e 

; 

T1 

aONVlUOdWI 

o 

(D 

~ 

2 

j 

1 Fast  Engineering  Prooete 

1 Reduce  # of  Prototypes 

1 

o 

S 

£ 

■5 

Faster  Tooling  Delivery 

Reduce  Cost  of  TooUng 

ABSOLUTE  IMPORTANCi 

Interaction  Points  from  Roof 

1 

•5 

2 

X 

i: 

(a  ^ 
^ X 
— in 
5<N 

L4- 


V 


LANGUAGES 

REOUlRE^^ENTS 

IMPORTANCE 

1 

on 

S 

§ 

IICN  1 

UhI 

LkJ 

LOTOS 

.nstantiation 

EXPRESS  Compatible 

9 

o 

Declore  and  Values  to  EXPRESS  Types 

9 

® 

0 

A 

A 

c 

Models  Must  Be  Self  Contained 

6 

® 

® 

Handle  EXPRESS  expressions 

6 

O 

A 

A 

® 

Save  and  Retrieve  Models 

'6 

A 

® 

A 

A 

0 

Manipulation 

Send  Commands 

'6 

o 

O 

Ai® 

o 

Describe  Commands 

'6 

'sj 

A 

aIa 

o 

Testing 

Specify  Admin  Informotion 

•6 

A 

® 

aIa 

c 

Test  Cose  Administration 

3 

A 

® 

A 

A 

o 

Specify  Test  Result  Evaluation 

9 

A 

® 

A 

A 

A 

Reoch  a Verdict 

9 

A 

® 

A 

A 

A 

Execute  Test  Based  on  Results 

'6 

A 

® 

A 

0 

A 

Reoch  Inside  the  Model 

'6 

A 

® 

A 

A 

Specify  Comparison  to  Reference 

'9 

O 

® 

A 

QIA 

High  Level 

Well  Defined 

6 

O 

A 

0 

® 

® I® 

Human  Unoerstondobie 

3 

o 

A 

® 

O 

a!a 

Mochine  Reodcbie 

9 

o 

® 

O 

® 

o 

® 

Simple 

3 

A 

A 

0 

o 

o 

C 

Mature 

3 

A 

o 

® 

® 

® 

Extensible 

6 

A 

O 

o 

A 

A 

® 

Supported  by  Tests  ond  Compii-ers 

9 

0 

0 

® 

® 

® 

Support  Executable  Tests  Automation 

9 

A 

O 

A 

® 

ABSOLUTE  importance 

r«» 

e>t 

, 

RELATIVE  IMPORTANCE 

R 

«« 

§ 

rsi 

^ MATRIX 

WEIGHTS^ 

Strong  ^ 

9 

Medium  Q 

3 

,Weok  /v 

’ J 

O 

< 

s 

u 

XI 

^ w 

^ § 

Q ^ 

S 
a 

Pi  w 
PQ 


Pi 

o 


C/D 

HH 

o 

HH 

H 

s 

Pi 

O 

z, 

HH 

H 

< 

X 


H 

04 

s 

o 

u 

pp 

Pi 

X 

Z 

o 

H 


CO 

C3 

lU 

u. 


<D 

O) 

C 

» 2 


CO  til 

is 


« 2 ^ 
CO  ^ 3 


- 

CO  ^ 

c O & 

O Ul  -O 
IL. 


E 

(0 

O 


m CM 
Q.—  a> 

|i? 

S-  = 

^ A 

= c 5" 

o :=  < 
o 

(0  o >» 

■O  C CO 

c ^ 
> O o 

-I* 

CO 

c 

o 

00 


o 

CO 

s 

a 


4k 

o 


■8 


Q 


OFEQSLtd  1991 


/ ESPRIT  CIM  PROJECT  21 95 

FAMexchange  architecture: 

separate  processors 


r 


CADEXi 


ESPRIT  CIM  PROJECT  21 95 

FAMexchange  architecture 

merged  processors 


CADEX 

J 


aSBB 


iBaiS? 


ESPRIT  CIM  PROJECT  2195 

CADEX  application  protocois  for  STEP 


rAMbulld  Date:  9 Mar  1992  Model:  TEAPOT 


o 

JC 

o 


o 

•a 

o 


CM 

a> 

0> 


it:::: 


t:::: 

t:::: 


• »> 


t: 


f::r 

[iHi 


::H 


i:::r4 
1 • • 

' I::*?’ 

' •-  • t ■ » 

! i 


' : A » 


I ' 


3 

U 


o 

1 

rr 

4- 

- rvi. 

O 

o 

ft«; 

A 

f 

U 

V 

[?4£ 

• • • — 

tijr 

H’'i 

■ 

rfi 

- 

t-'^i 

X 

<D 


a-  !►>.„ 

(n 

it 


W 

0^:0-  U U O »-  (/) 


o 

o 


o 

o 


OJ 

a> 

O) 


o 

o 


3 


rANbulld  Date:  12  Feb  1992  Model:  shell_51 


u 

z 

hH 

03 

u 

Q 

CL, 

QC 

O 

b 

cc 

O 

o 

s 

H 

s 

z 

o 

S 

S 

o 

u 


Steve  Ryan 

Core  Mechanical  Project  - PDES,  Inc. 
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I ne  Key  cnteria  used  to  make  such  a decision  is  whether,  and  to  what 
degree,  the  resource  areas  required  have  been  previously  tested. 
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Kent  Reed  NIST  (on  behalf  of  NIDDESC) 

Member  of  Navy  Industries  Digital  Data  Exchange  Standards  Committee 
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How  developed: 


Application  Reference  Model 
How  developed: 
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Part  owner’s  word  processor  (drawings) 
NIAM  tool  repositories  (files,  DB) 


ARM-AIM  Mapping  TahiA 
How  developed: 

• Drafted  bv  oart  owner 
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• Part  owner’s  word  processor  (will  include  EXPRESS-G  drawings) 


(Conformance  Requirements  and)  Test  Purposes 
How  developed: 
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The  Roles  of  Mapping  Tables 
and  Conformance  Test  Purposes 
in  STEP  Application  Protocols 
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Julian  Fowler  ^ C-'cO 
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Agenda 

• Background 

• What  is  an  AP? 

• Relationships  between  components  of  an  AP 

• The  role  of  the  mapping  table 

• Contents  of  the  mapping  table 

• current  position 

• future  requirements 

• Conformance  test  purposes 

• what  they  are 

• how  they  are  developed 

• how  they  are  documented 

• Relationship  between  mapping  table  and  conformance  test 
purposes 

• Usage  test  purposes 
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Background 

• Mapping  tables  and  test  purposes  have  been  required 
components  of  STEP  Application  Protocols  since  the  AP  was 
adopted  as  part  of  STEP  in  1989 

• Active  work  on  APs  has  to  date  concentrated  on  AAMs, 

ARMS  (the  AP  developers),  AIM  development  and  integration 

• Approaches  to  mapping  tables  and  test  purposes  have  not  been 
agreed  or  documented:  note  the  differences  between  Parts 

201  and  203 

• The  key  role  of  the  mapping  table  has  emerged  in 
consideration  of  how  test  purposes  relate  to  the  AP  and  to 
the  abstract  test  suite 

• This  presentation  summarises  the  current  position  and 
proposes  how  we  should  move  forward 

d«ni\82ooia 
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What  is  a STEP  Application  Protocol? 


• A STEP  AP  Is  a Standard 

• A STEP  AP  is  a specification  of  data  constructs  (entities 
€md  structure)  required  to  exchange  and/or  share  product 
information  to  meet  a documented  industrial  requirement 

• A STEP  AP  may  specify  several  "levels”:  each  of  these  could 
be  documented  as  a separate  AP  Part 

• A STEP  implementation  combines  a STEP  AP  and  an 
implementation  form  (physical  file,  SDAI,  etc) 

• A STEP  AP  is  not  a subset  of  the  STEP  Integrated  Resources 

• STEP  APs  are  to  be  integrated  to  enable  interoperability 
between  implementations  of  diffferent  APs  that  share  common 
information 

• "Application  Interpreted  Constructs"  specify  potentially 
shareable  information  structures 
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STEP  AP  Interoperability 
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Relationship  between  AP  components 
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AlCs  as  complete,  testable  "mini  APs" 
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The  role  of  the  mapping  table 


• The  mapping  table  documents  the  correspondence 
between  the  information  requirements  and  the  AIM.  The 
listing  shall  provide  a complete  and  unambiguous  mapping 
between  the  constructs  defined  in  clause  4 and  the 
constructs  defined  in  the  AIM,  including  preservation  of 
the  construct  assertions." 

(Guidelines  for  the  development  and  approval  of  STEP 

Application  Protocols,  version  1 .0,  WG4  N34) 

• The  mapping  table  provides  an  explicit,  normative  link 
between  the  application  domain  and  its  language,  and  the 
"STEP"  domain  and  its  language. 

• The  level  of  detail  and  length  of  the  mapping  table 
depends  directly  upon  the  level  of  detail  to  which  the 
Information  requirements  are  modelled. 
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Contents  of  the  mapping  table 


• In  developing  Part  201  a two-level  mapping  table  has  been 
employed.  The  content  and  layout  employed  has  not  yet  been 
agreed  from  either  the  technical  or  editorial  viewpoint. 

• Each  ARM  construct  (entity)  ie  mapped  to  a "primary" 
construct  in  the  AIM. 

• This  level  may  be  sufficient  for  APs  where  the  mapping  is 
relatively  simple,  e.g.  the  mappings  from  the  geometric 
requirements  In  Parts  204, 205  and  206  to  the 
geometry„8chema  in  Part  42 

• Where  this  "primary"  mapping  is  not  complete  and 
unambiguous,  each  attribute  of  the  ARM  entity  is  mapped  to 
one  or  more  AIM  constructs;  in  each  case  a unique  "reference 
path"  must  be  stated. 

• Any  and  all  global  rules  in  the  AIM  that  play  a role  in 
satisfying  the  ARM  requirement  are  stated  explicitly. 
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Future  requirements  for  mapping  tabies 


• Current  approach  results  in  a mapping  table  that  is  only 
useful  for  determining  a "one  way"  mapping:  ARM  >>  AIM; 
consideration  of  test  purposes  suggests  that  a two-way  table 
(or  tables)  may  be  required;  i.e.,  given  an  AIM  construct, 
which  ARM  construct(s)  does  it  play  a role  in  satisfying? 

• The  mapping  table  must  satisfy  the  need:  "what  query  or 
queries  on  an  implementation  of  the  AIM  will  give  all  the 
data  necessary  to  populate  a CAx  system  database  instance 
corresponding  to  the  given  ARM  construct?" 

• Mapping  tables  should  be  computer-processable. 

• is  the  above  requirement  synonymous  with,  and/or  dependent 
on,  a need  for: 

• standardised,  formal  method(s)  for  documenting 

requirements; 

• a standardised  mapping  language  use  in  the  interpretation 
process? 


Conformance  test  purposes 


• A conformance  test  purp>ose  is ...  ”a  precise  description 
of  the  objective  which  an  abstract  test  case  is  then 
designed  to  achieve.” 

(iSO  CD  10303-31  "Conformance  Testing  Methodology  and 
Framework") 

• A conformance  test  purpose,  taken  together  with  the 
aspect  of  the  mapping  table  exercised  by  the  test  purpose 
and  the  corresponding  parts  of  the  information 
requirements  and  AIM,  is  a specification  of  an  abstract 
test  case. 

• Conformance  test  purposes  identify  all  the  discrete 
options  in  the  AIM:  OPTIONAL  attributes,  subtypes,  select 
types,  enumerations  (including  BOOLEAN  and  LOGICAL) 
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Documentation  of  conformance  test  purposes 


• Test  purpose  name  is  that  of  the  corresponding  AIM  entity. 

• The  context(s)  in  which  the  test  purpose  exists  are  listed; 
these  are  all  the  roles  that  the  AIM  entity  plays. 

• All  discrete  options  in  the  entity  are  listed,  using  a 
standard  language. 

• TEST_PURPOSE - 
ENTITY.NAME 
(•’AS-ENTITY_TYPE)| 

•WITH"  ATTRIBUTE.NAME  ATTRIBUTE_OPTIONS ) . 

• ATTRIBUTE_OPTIONS - 
■NOT  PRESENT"  | 

I "PRESENT"] 

[ "HAVING  AT  LEAST  ONE  ELEMENT  [ "PRESENT  ] ] 

( "AS"  ENTITY_TYPE  1 "-"  ENUMERATION.VALUE" ) . 
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Relationship  between  mapping  tabie  & test  purposes 


• We  know  Intuitively"  that  there  is  a close  relationship 
between  the  mapping  table  and  the  test  purposes. 

• An  algorithm  exists  for  the  generation  of  test  purpose 
"syntax"  from  the  AIM  EXPRESS  schema  (this  has  not,  as  yet, 
included  consideration  of  the  effect  of  local  and  global 
rules). 

• The  future  requirements  for  Improved  mapping  tables  Include 
better  Identification  of  the  AIM  >>  ARM  mapping. 

• Given  such  an  extended  - and  computer  processable  • 
mapping  table,  it  may  be  the  case  that  the  conformance  test 
purposes  are  generated  automatically  In  the  development  of 
the  abstract  test  suite,  and  may  not  be  required  as  part  of 
the  AP  documentation. 


Usage  test  purposes 


• In  addition  to  conformance  test  purposes,  additional  tests 
which  exercise  specific,  and  possibly  complex,  combinations 
of  data  in  the  AP  may  be  identified. 

• These  test  purposes  are  likely  to  result  from  AP  (and 
specifically  ARM)  validation  exercises. 

• Such  test  purposes  are  likely  to  be  used  in  assessing 
specific  CAx  systems  as  potential  candidates  for 
Implementation  of  the  AP. 

• Such  test  purposes  are  included  in  clause  6 of  Part  203  as 
"complex  test  purposes". 

• Recommendation:  usage  test  purposes  should  be  an  optional 
element  of  Annex  G of  the  AP  ("AP  Usage  Guide"). 
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To  : Participants  in  the  AP  Validation  Workshop  April  13'14  1992, 

ISO/IPO  meeting  In  Seattle.  April  1992 

From  : Jon  Aas.  PEGS  Ltd 

Telephone:  +aa  223  237111 
Fax:  •f44  223  234192 

Email: 

Position  paper:  *'AP  deveiopment  procedures  and  formai 
milestones,  will  we  ever  get  to  the  end?** 

1.  Introduction 

The  methodology  defining  the  application  protocol  development  process  has  been  progressing 
very  rapidly  over  the  last  year.  The  current  version  of  the  AP  Guidelines,  ISO  TC184/SC4/WG4 
N32  (P5),  dated  6 January  1992,  sets  out  the  structure  of  the  work,  defining  a number  of 
activities  and  milestones  to  be  reached  in  the  process  of  developing  an  AP  within  ISO. 

For  anyone  who  wants  to  propose  an  AP  project  within  ISO,  it  is  necessary  to  estimate  the  man 
time  needed  to  reach  the  different  milestones.  There  is  no  indications  as  to  when  the  different 
stages  in  the  AP  deveiopment  should  be  completed,  measured  in  terms  of  ISO  meetings  or  hours 
of  Validation,  Qualification,  Editing,  etc.  Committee  effort.  In  other  words,  the  amount  of 
resources  needed  to  complete  an  AP,  ie  bring  it  up  to  a DIS  status  is  unknown  or  open-ended. 

This  position  paper  will  describe  the  problem  and  suggest  a possible  strategy  for  the  completion 
of  AP  projects,  based  on  the  AP  Guidelines  document  and  the  information  I have  available  on  the 
AP  Qualification  Manual  (I  am  waiting  for  an  update). 

2.  The  Goal 

The  target  is  to  pin  down  a policy  within  WG4/P1  & P5  that  will  allow  AP  project  leaders  to 
estimate  the  time  needed  to  complete  the  work  and  to  monitor  the  progress.  This  will  also  allow 
the  Qualification  and  Validation  project  to  structure  its  work  so  it  is  possible  to  handle  the 
avalanche  of  APs  that  is  expected  as  soon  as  STEP  Version  1 .0  is  out. 

3.  The  current  structure  of  the  Process  for  Developing  and 
Qualifying  a STEP  Application  Protocol 

The  current  structure  is  well  defined  and  serves  some  well  defined  purposes: 

• it  ie  incremental  to  ensure  that  commonality  between  different  APs  are  identified  at  an  early 
stage 

• it  is  incremental  to  allow  a comprehensive  review  procedure  at  each  stage  in  the  AP 
deveiopment  process 

The  incremental  nature  of  the  process  will  provide  APs  of  high  quality. 

The  components  of  an  AP  are  defined.  The  development  of  an  AP  consists  of  the  development  of  a 
succession  of  components,  where  each  step  in  the  deveiopment  process  builds  upon  the  precision 
and  documentation  of  the  previous  steps. 

Each  component  of  an  AP  proceeds  through  three  basic  steps: 

1)  define  the  requirements  and  evaluation  criteria  for  the  component 

2)  develop  the  component 

3)  exercise  the  criteria  to  evaluate  the  component 

The  AP  Deveiopment  and  Approval  process  is  defined  in  three  strands: 

• the  AP  Development  Process 

- the  AP  Validation 

- the  AP  Review  and  Qualification  Process 
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They  are  Interlinked,  as  the  AP  Review  and  Qualification  Process  wilt  start  when  the  AP 
Development  la  still  in  progress.  The  AP  Review  and  Qualification  Process  has  two  phases.  A and 
B. 

Phase  A:  AP  Development  Reviews,  for  reviews  while  the  AP  components  are  being  developed. 
Phase  B:  WG4/QP  AP  Qualification  when  the  AP  is  complete 

AP  Review  and  Qualification  Process 


4.  Contributors  to  AP  Development  Projects 

There  are  many  contributors  to  the  process  of  developing  an  AP.  This  is  the  list  of  all 

contributors  and  their  responsibilities: 

Industry  Representatives  documents  requirements  for  product  data  communication  and 

potential  APs.  produce  a Candidate  AP  Summary 

STEP  Experts  Identify  correspondence  between  industrial  requirements  and  scope 

and  architecture  of  STEP. 

The  AP  Project  Team  develops  the  AP  itself  and  documents  it 

SC4PMAQ  approves  an  AP  development  project.  They  also  monitor  the 

progress  of  AP  projects  and  provides  oversight  coordination  and 
resource  allocation. 

WG4  AP  Integration  Project  reviews  and  evaluates  the  ARM.  UoF  and  information  requirements 

WG4  AIM  Development  Project  shall  provide  technical  advice  and  reviews  for  AP  projects. 

WG4  STEP  Part  Qualification  Project 

shall  provide  technical  advice  and  reviews  for  AP  projects. 

WQ4  STEP  Part  Qualification  Project  Leader 

is  responsible  for  running  the  WG4  activities  to  AP  completion 

W06  reviews  and  approves  conformance  requirements,  test  group 

structure  and  test  purposes 


5.  Allocation  of  rasoureaa 

The  AP  Project  Leader  is  the  one  to  define  the  project  duration  and  allocate  resources  to 
different  activities. 

The  AP  project,  through  its  Project  Leader,  is  responsible  for  requesting  resources  from  WG4 
AIM  Development  Project  and  WG4  STEP  Part  Qualification  Project  for  meetings  during  the 
development  period. 

6C4  PMAG  will  approve  the  work  plan  and  subsequently  commit  the  necessary  resources  to 
support  the  AP  Development  and  Approval  Process. 


6.  Milestones  and  Scheduled  Meetings 

There  is  a vast  number  of  meetings  that  are  scheduled  from  the  start  to  the  end  of  an  AP  project 
Here  is  a list  of  meetings  given  in  the  chronological  order,  indicating  who  should  be  present. 
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6.1  Thd  AP  Development  Process 

A number  of  meetings  are  required  for  the  industry  speclaiists  to  document  requirements  for 
product  data  communication  and  potential  APs.  No  estimates  in  time  and  effort  Is  made. 

The  AP  Project  Team  should  conduct  a set  of  industry  reviews  and  evaluations  of  the  AP  scope 
and  requirements.  No  estimates  In  time  and  effort  is  made. 

A lot  of  home  work  has  to  be  done  by  the  AP  Project  Team.  This  will  depend  on  the  area  addressed 
by  the  AP  and  the  time  available  for  In  depth  analysis.  No  estimates  in  time  and  effort  is  made. 

A number  of  meetings  are  scheduled  together  with  the  WG4/STEP  Part  Qualification  Project  and 
other  contributing  to  the  process. 

6.2  The  AP  Validation  Procett 
A number  of  meetings  are  listed. 

6.3  The  AP  Review  and  Qualification  Procese 

Phase  A;  AP  neveiopment  Reviews 


Activity  A1 
Activity  A2 
Activity  A3 
Activity  A4 
Activity  A5 
Activity  A6 
Activity  A7 
Activity  A8 


WG4/STEP  Part  Qualification  Project  & AP  Project  Team 

WG4/AP  Integration  Project 

WG4/AiM  Development  Project 

WG4/STEP  Part  Qualification  Project 

WG4/AIM  Development  Project 

WG4/STEP  Part  Qualification  Project 

WG6 

AP  Project  Team,  WG  convener 


EhaSfi  B; WG4/STEP  Part  AP  Qualification 


Activity  B1 
Activity  B2 
Activity  B3 
Activity  B4 
Activity  B5 
Activity  B6 
Activity  B7 
Activity  B8 


WG4/STEP  Part  Qualification  Project  Leader 
AP  Project  Team 

WG4/STEP  Part  Qualification  Project  Leader 
WG4/AP  Qualification  Panel  & AP  Project  Team  membor{s) 
WG4/STEP  Part  Qualification  Project  Leader  on  his  own 
WQ4/AP  Qualification  Panel  members  off-site 
WG4/AP  Qualification  Panel  & AP  Project  Team 
WG4/STEP  Part  Qualification  Project  Leader  on  his  own 
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7.  The  document  exchange  for  an  AP  in  ISO 


The  entire  AP  Development  and  Approval  Process  is  based  on  a continuous  exchange  of 
documents.  The  documents  are  produced  by  several  of  the  contributors  in  this  game,  most 
notably  the  AP  development  team  represented  by  the  AP  project  leader.  The  structure  of  the 
process  is  here  defined  in  terms  of  document  exchange. 


7.1  The  AP  Development  Procete 

Activity  1 : The  result  is  a document  describing  the  Industrial  requirements  for  an  AP  or  a 
suite  of  APs. 

Activity  2:  The  results  is  an  initial  scope  definition  of  an  AP  or  a suite  of  APs. 

Activity  3:  A candidate  AP  Summary  submitted  to  SC4  PM  AG.  Input  to  SC4  PM  AG  from  WG2, 

WQ3.  W04 


Activity  4: 
Activity  5: 
Activity  6: 
Activity  7: 

Activity  0: 

Activity  9: 


Scope  and  Requirements  Evaluation  Report,  to  go  into  the  AP  Validation  Report. 
AP  Development  Plan,  including  the  AP  Validation  Plan. 

The  definition  of  the  ARM  and  UoFs 

Report  on:  Information  Requirements,  ARM  & UoFs  submitted  to:  WG4  AIM 
Development  Project 

Report  on:  Information  Requirements,  ARM  & UoFs  submitted  to:  WG4  AP 
Integration  Project 

Report  on:  ARM  Validation  Report,  a part  of  the  AP  Validation  Report.  AP  Usage 
Tests 


Activity  10: 

Activity  11: 
Activity  12: 
Activity  13: 
Activity  14: 

Activity  15: 

Activity  16: 


Report  on:  Group  1 components  submitted  to:  WG4  STEP  Part  Qualification 
Project 

Report  on:  AlC  Library  and  integrated  Resources  mapping  to  the  AIM. 

Report  on:  ARM  to  AIM  Mapping,  a part  of  the  AP  Validation  Report 

Report  on:  AIM  validation  Report,  a part  of  the  AP  Validation  Report 

Report  on:  Group  2 components  submitted  to:  WG4  STEP  Part  Qualification 
Project 

Report  on:  AP  Conformance  Requirements,  Test  Group  Structure  and  Test 
Purposes,  a part  of  the  AP  Validation  Report 

Report  on:  Completed  AP  document  submitted,  AP  Issues  Log  and  AP  Validation 
Report 


7.2  The  AP  Validation  Process 

The  AP  Project  Team  should  produce  an  AP  Validation  Plan.  The  AP  Validation  Plan  shall  be 
review  with  the  WG4  AP  Guidelines  Project  and  WG6.  The  AP  Validation  Plan  and  the  resulting 
AP  Validation  Report  shall  be  submitted  with  the  complete  Draft  AP  for  review  and  acceptance 
by  the  WG4  Qualification  Project. 

The  components  are  described  in  the  previous  section. 
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7.3  Thd  AP  Review  and  Qualification  Proceae 

The  documents  to  be  produced  in  the  AP  Review  and  Quaiification  Process  are  as  foiiows: 

Phase  A:  AP  Deveiopment  Reviews 

Activity  A8:  The  Draft  AP  is  is  submitted  for  approval  by  WG4/5TEP  Part  Qualification 
Project. 

Phase  B:  WG4/STEP  Part  Qualirication  Project  AP  Qualification 

Activity  B2:  A possible  undated  AP  to  be  distributed  to  WQ4/STEP  Part  Qualification  Panel 
members 

Activity  B3.1 : WG4/STEP  Part  Qualification  Project  Leader  develops  workshop  plan  and 
schedule. 

Activity  B5:  WG4/STEP  Part  Qualification  Project  Leader  prepares  a preliminary 
qualification  report 

Activity  B6:  WG4/STEP  Part  Qualification  Project  Leader  completes  the  AP  qualification 

report  and  submits  the  report  to  the  AP  Project  Team  Leader.  Editing  Committee  and  SC4  PMAG. 


8.  Discussion 

The  current  structure  of  the  AP  Guidelines  document  is  such  that  the  effort  required  to  achieve 
an  approved  AP  is  scattered  around  in  the  document.  No  effort  has  been  made  to  quantify  the 
resources  needed  to  complete  the  job.  or  how  it  should  be  possible  to  get  the  various  groups  to 
contribute  their  part. 

It  is  necessary  to  get  the  resource  requirements  defined  and  a possible  work  schedule  identified. 
For  anyone  doing  project  management  of  an  AP  project,  it  is  irrportant  to  have  this  information 
readily  available  for  the  planning  and  costing  of  a project. 

The  AP  Project  Leader  must  be  able  to  forecast  the  completion  of  the  project,  and  base  his 
estimates  on  the  availability  of  necessary  resources.  The  AP  Project  Plan  is  among  the  first 
documents  an  AP  Project  Leader  has  to  produoe  for  SC4  PMAG  approval. 

The  scheduling  should  be  based  on  the  ISO  and  iSO/iPO  meeting  schedule,  knowing  that  the 
project  will  get  the  necessary  resources  made  available  to  them  at  meetings  or  in  the  Interim 
period  to  progress  work  according  to  a predefined  work  plan. 

With  such  a structure  of  the  individual  AP  projects.  It  will  be  possible  to  predict  the  need  for 
meetings  for  the  AP  Quaiification  Project  Leader  and  to  allocate  the  necessary  resources  to  the 
activities.  This  is  absolutely  necessary  to  be  able  to  cope  with  the  expected  avalanche  of  APs 
following  the  release  of  STEP  Version  1 .0. 
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9.  Recommendations 

It  is  Important  to  get  a structure  in  the  AP  Development  and  Approval  Process.  The  following 
recommendations  are  made: 

Recommendation  1:  It  Is  necessary  to  go  through  the  AP  Development  and  Qualification 
Process  to  rationalise  the  milestones  and  the  work  effort.  It  must  be  a target  for  the  WG4 
Guidelines  Project  to  define  a work  schedule  that  will  enable  an  AP  Project  to  go  through  the 
process  from  start  to  finish  within  5-6  ISO  meetings,  le  less  that  18  months. 

Recommendation  2:  A recommended  work  plan  should  be  developed,  to  give  AP  projects  a 
guide-line  for  what  is  required  in  time  and  effort  to  make  a DIS  AP.  This  work  plan  should 
contain  all  milestones  for  completion  of  the  different  activities,  tied  down  to  named  ISO  and 
ISO/IPO  meetings. 

Recommendation  3:  Create  a set  of  formal  request  forms,  which  each  AP  Project  Leader  Is 
expected  to  use  when  requesting  response  from  SC4  PM  AG  or  any  other  contributor  to  the 
process 

Recommendation  4:  Get  the  required  man  power  made  available  to  the  WG4/STEP  Part 
Qualification  Project.  This  should  no  longer  be  based  on  voluntary  contributions,  but  be  fatly 
•*paid-conBoltanoy  work: 

Recommendation  5:  Nothing  has  been  defined  for  the  case  of  unsuccessful  reviews,  ie  some  of 
the  activities  have  to  be  repeated.  The  problem  of  reducing  the  delay  In  iterations  has  to  be 
addressed. 


10.  Open  Issues 

There  are  still  a number  of  questions  to  be  answered  about  the  entire  process.  Here  are  a few: 

- Is  it  possible  to  simplify  the  process  of  AP  Development  and  Approval? 

- What  happens  with  an  AP  effort  that  has  not  followed  the  process  assumed  in  the  AP  Guidelines 
document,  but  has  for  instance  been  developed  to  corrpletion  outside  ISO? 
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TIm  Use  of  Application  Model  Validation  in  Testing  a Proposed  Standard 

Mary  J.  Mitchell  and  Katherine  C.  Morris 
National  institute  of  Standards  and  Technology 
Building  220,  Room  A127 
Gaithersburg,  MD  20899 
(301)975-3538 
FAX  (301)  258-9749 
email:  mitchell@cme.nistgov 

Abstract 

An  Application  Protocol  (AP)  is  a specification  of  data  sharing  requirements  for  a particular  application  area.  The 
Standard  for  the  Exchan^  of  Product  Model  Data  (STEP)  provides  an  integrated  collection  of  product  definitions  which 
allow  for  the  unambiguous  description  of  application  data  requirements.  Application  Protocols  are  designed  to  permit 
practical  implementations  of  STEP.  However,  validation  testing  is  needed  to  ensure  that  the  technical  solutions  provided 
by  an  AP  will  work  in  a practical  sense.  This  validation  focuses  on  the  principal  mechanism  for  specifying  the  data 
sharing  requirements,  an  application  specific  information  model.  The  b^y  of  the  paper  describes  the  process  by  which 
an  application  model  is  validated. 

Application  model  development  and  validation  is  a complex  process  that  relies  extensively  on  human  capabilities  for 
analysis,  judgement  and  synthesis  of  large  amounts  of  diverse  information.  This  process  requires  the  support  from 
automation  to  produce  a technically  complete  AP.  The  model  validation  process  and  the  STEP  development  methods 
place  unique  requirements  on  the  software  that  will  be  needed  to  support  the  effective  testing  of  STEP.  The  National 
PDES  Testbed  at  the  National  Institute  of  Standards  and  Technology  has  undertaken  a software  project  to  support  this 
process.  The  current  direction  of  this  project  has  been  formulated  from  our  initial  experiences  in  exercising  this  process 
and  with  software  automation  for  the  model  validation  testing.  In  addition,  this  paper  introduces  the  potential  contribution 
that  application  model  validation  and  validation  tools  could  make  to  the  conformance  testing  of  AP  implementations. 
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The  Use  of  Application  Model  Validation  in  Testing  a Proposed  Standard 

1 Introduction 

From  an  information  system  development  perspective,  logical  data  modeling  techniques  have  traditionally 
served  in  two  roles.  The  first  role  is  as  a method  of  describing  the  information  requirements  of  an 
application  system.  The  second  role  is  as  a mechanism  for  integrating  the  requirements  from  a number 
of  applications  into  a single  logical  and  consistent  schema  so  that  data  can  be  shared  by  multiple 
applications.  The  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  project  is  developing  an 
international  standard^  which  uses  data  modeling  as  the  basis  for  a multi-national  and  multi-enterprise 
integration  effort  STEP  is  designed  to  provide  a complete,  unambiguous,  computer-readable  definition  of 
the  characteristics  of  a product  throughout  its  life  cycle.  STEP  product  definition  specifications  are 
implementation  independent  though  implementation  interface  techniques  provide  the  communication 
mechanisms  for  applications  using  file  exchanges^  or  shared  databases.  Because  of  the  diversity  of 
applications  that  are  within  the  scope  of  STEP,  the  integration  generally  causes  extensive  changes  but 
the  changes  are  justified  if  all  of  the  information  requirements  are  supported. 

This  paper  describes  a method  for  ensuring  that  all  application  information  requirements  are  met.  While 
much  of  the  terminology  is  specific  to  the  development  methods  used  for  STEP  [Danner92,Palmer91], 
any  large  scale,  multi-enterprise  integration  activity  should  find  this  material  useful.  The  method  for 
validating  a logical  data  model  should  be  applicable  even  if  the  data  modeling  technique  is  other  than  one 
of  those  used  by  STEP. 

Confidence  in  a standard  by  its  user  community  is  absolutely  essential  for  a standard  to  gain  acceptance. 
Validating  that  user  needs  are  supported  by  the  proposed  standard  is  the  foundation  for  any  useful 
standard.  Similarly,  confidence  that  an  integrated  data  model  supports  user  needs  is  essential  in  any 
implementation  of  any  large,  complex  database  system.  Significant  investments  will  be  required  to 
Implement  STEP  applications.  Because  of  the  integration,  STEP  will  contain  a number  of  untried 
solutions  to  technical  problems.  The  STEP  user  community  must  not  ask  vendors  to  develop 
implementations  based  on  specifications  that  have  unknown  levels  of  risk.  Thorough,  appropriate  model 
validation  testing  before  standardization  can  minimize  this  risk.  In  addition,  this  testing  will  reduce  the 
need  to  continually  ’patch’  the  standard  to  correct  design  flaws  uncovered  by  implementing  the  standard. 
The  use  of  proven  integrated  data  models  provide  the  mechanism  for  controlling  these  risks.  Developers 
of  large  integrated  databases  should  want  the  same  sort  of  proof  that  their  integrated  logical  data  model 
is  correct  prior  to  implementing  numerous  applications. 


' The  International  Organization  for  Standardization  (ISO)  is  developing  ISO  10303  - Industrial 
Automation  Systems:  Product  Data  Representation  and  Exchange  in  Technical  Committee  184  (TC  184) 
Subcommittee  4 (SC4)  on  Industrial  Data  and  Global  Manufacturing  Programming  Languages.  For  an 
overview  of  ISO  10303  refer  to  Part  1:  Overview  and  Fundamental  Principles  [ISOIJ. 

^ The  initial  release  of  STEP  supports  only  a file  exchange  interface  but  an  interface  for  exchange 
through  database  transactions  is  under  development. 
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An  Application  Protocol  (AP)  is  a specification  for  a portion  of  the  product  data  described  by  STEP 
[PalmerSI].  APs  are  the  pairts  of  STEP  that  are  to  to  directly  implemented  by  vendors  of  software 
products  who  adopt  STEP.  Tbe  entirety  of  STEP  consists  of  all  STEP  APs  along  with  supporting 
definitions  for  the  elements  that  are  common  to  many  APs,  the  specification  languages,  implementation 
interfaces,  and  conformance  testing  requirements.  AP  specifications  are  derived  from  a set  of  integrated  ‘ 
data  models,  called  Integrated  Resources’,  that  contain  product  definitions  in  a broad  context,  e.g. 
across  the  product  life  cycle  and  across  manufacturing  disciplines.  In  the  AP,  these  general  descriptive 
definitions  for  products  are  interpreted  to  describe  specific  data  requirements  for  a given  application. 
Within  an  AP  there  is  no  redundancy  and  there  is  consistency  with  the  integrated  resources.  Due  to  this 
consistency,  the  STEP  AP  specifications  permit  product  information  to  to  unambiguously  exchanged  or 
shared  between  implementations  on  dissimilar  sterns. 

Procedures  are  needed  to  ensure  that  the  technical  solutions  provided  by  STEP  APs  and  integrated 
resources  will  work  in  a practical  sense.  The  term  application  model^  is  used  throughout  this  paper  to 
refer  to  the  component  information  models  in  an  AP  or  any  other  domain  specific  information  model  with 
similar  properties.  In  1991,  Mitchell  [Mitch91]  proposed  a methodology  for  validating  STEP  AP  models. 
The  National  PDES  Testbed^  is  used  to  test  the  validity  of  application  models  and  the  software  from  its 
Validation  Testing  System  (VTS)  supports  this  methodology.  This  document  introduces  the  methodology 
and  shows  how  the  VTS  applies  the  proposed  methodology  to  support  AP  model  validation.  The 
methodology  and  software  build  on  previous  experience  with  testing  STEP  from  an  application's 
perspective.  Another  type  of  testing,  the  conformance  testing  of  vendor  AP  implementations,  should 
leverage  the  VTS  as  well  as  some  of  the  outputs  from  application  model  testing. 

A detailed  discusion  of  the  VTS  can  to  found  in  a series  of  reports  from  the  National  PDES  Testbed 
[Mitch90,Mitch91,Morris91a,Morris91b,Morris91c].  Section  2 of  this  document  gives  an  overview  of  the 
validation  testing  process  for  APs.  Section  3 describes  the  activities  which  comprise  the  validation  testing 


* The  initial  Integrated  Resources  in  STEP  include:  Integrated  Resource  Fundamental  Concepts, 
General  Shape  Representation  Concepts  (which  includes  geometry  and  topology).  Representation 
Structures,  Product  Structure  Ck^nfiguration  Management,  Presentation,  and  General  Draughting  (k>ncepts. 
For  a technical  description  refer  to  Part  41:  Integrated  Generic  Resources  - Fundamentals  of  Product 
Description  and  Support  [IS041]. 

* The  information  models  that  are  components  of  an  AP  are  called  application  interpreted  models 
(AIMS)  and  application  reference  models  (ARMs).  Other  domain  specific  information  models  include 
context-driven  integrated  models  (CDIMs)  which  are  used  to  evaluate  STEP  resource  models  [CDIM  A1] 
and  various  AP  precursor  models  existing  outside  of  the  standardization  process  for  purposes  such  as 
vendor  prototyping.  The  validation  methodology  is  applicable  to  these  other  application  models  If  STEP 
development  methods  are  used.  The  first  priority  of  the  VTS  project  is  to  support  the  requirements  for 
validation  of  AIMs  and  CDIMs. 


' The  National  PDES  Testbed  is  located  at  the  National  Institute  of  Standards  and  Technology.  Funding 
for  the  Testbed  Project  has  been  provided  by  the  Department  of  Defense's  Computer-Aided  Acquisition  and 
Logistic  Support  (CALS)  Office.  The  work  described  in  this  document  is  funded  by  the  United  States 
Government  and  is  not  subject  to  copyright. 
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methodology  [Mitch91].  The  last  section  describes  the  future  directions  for  the  VTS  software,  based  on 
experiences  with  an  interim  software  system  used  at  the  National  POES  Testbed.  These  experiences 
provide  a basis  for  the  VTS  software  architecture  descn'bed  by  Morris  [Morris91c].  Rnally,  the  overlap  of 
software  requirements  in  the  validation  of  application  models  and  in  the  conformance  testing  of  vendor  AP 
implementations  is  discussed. 


2 Overview  of  Validation  Testing 

Application  protocol  development  and  testing  is  a complex  process.  It  involves  the  synthesis,  analysis, 
and  manipulation  of  large  amounts  of  diverse  information.  Most  of  the  process  relies  exclusively  on 
human  capabilities  for  analysis,  judgment,  and  interaction;  however,  part  of  this  process  can  and  should 
be  automated.  The  strategy  for  automation  is  based  on  an  analysis  of  the  information  flow  of  the  AP 
development  and  testing  process  and  initial  experiences  with  automation  for  validation  testing  at  the 
National  PDES  Testbed.  This  section  describes  the  validation  process  In  general.  For  a more  detailed 
presentation,  refer  to  the  proposed  AP  model  validation  methodology  [Mitch91]. 

Validation  testing  of  AP  information  models  determines  whether  the  AP  does  what  it  is  intended  to  do, 
i.e.,  meets  the  functional  requirements  that  led  to  its  development  The  integrated  resources  of  STEP  are 
also  shown  to  be  capable  of  supporting  the  application  area  The  proposed  approach  is  to  validate  AP 
models  by  simulating  the  behavior  of  relevant  applications  using  indu^  contributed  data.  The  validation 
tests  are  identified  by  examining  the  application  processes.  The  types  of  data  required  to  perform  each 
activity  in  the  application  process  are  specified  in  detail.  Realistic  data  from  the  application  domain  is 
associated  with  each  of  these  tests.  Multiple  sets  of  data  may  be  used  to  ensure  that  the  expected  range 
and  variation  of  industry  uses  can  be  supported  by  the  application  model.  The  data  needed  to  perform  a 
specific  process  or  generated  by  a specific  process  is  then  mapped  into  the  structures  defined  by  the 
application  model.  This  approach  to  validation  essentially  simulates  the  behavior  of  an  application  system 
interacting  with  the  file  or  database  system  that  provides  data  storage.  Since  an  AP  is  used  for  data 
sharing,  its  performance  must  be  validated  against  the  data  access  requirements  for  the  application. 

The  development  and  validation  of  an  application  model  can  be  decomposed  into  the  following  seven 
high-level  activities^  Some,  activities  may  be  performed  by  separate  groups  of  people.  The  first  two 
activities  establish  the  application  context  and  construct  application  data  models  that  will  be  tested.  The 
next  three  activities  focus  on  the  application  model  testing,  three  through  five,  evaluate  the  correctness  of 
the  application  models.  Activity  6 controls  the  identification  and  resolution  of  issues  against  the  model. 
These  issues  must  be  resolved  to  assure  confidence  in  the  model.  Once  the  application  models  have 
been  validated,  the  remaining  AP  components  can  be  developed.  In  the  seventh  activity,  the 
requirements  are  defined  for  what  in  implementation  must  be  tested  for  conformance  to  the  standard.  All 
resulting  outputs  which  are  required  components  of  an  AP  are  listed  in  italics. 


• For  a discussion  which  focuses  on  AP  development  and  AP  project  planning,  refer  to  Development 
Plan:  Applicarton  protocols  for  Mechanical  Parts  Production  [Stark91J. 
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L Scoping  the  Application  Context 


The  activity  of  "Scoping  the  Application  Context"  identifies  a formal  technical  boundary  for  the  application 
model  by  examining  the  functions  of  the  application.  The  boundary  is  defined  by  analyzing  the  general  . 
processes  the  application  performs  using  an  activity  model.  The  activity  model,  which  illustrates  the 
scope  of  the  application  area,  is  reviewed  by  experts  in  the  area  to  ensure  that  it  reflects  common 
business  practices.  The  scope  and  requirements  guide  the  determination  of  what  needs  to  be  tested. 

The  results  of  the  activity  follow: 

• an  activity  model  represented  in  IDEFO  which  defines  the  application  processes  in  the  AP,  and 

• a statement  of  scope  and  overail  requirements 

IL  Model  Construction 

During  the  "Model  Constnjction"  activity  detailed  information  models  are  constructed,  interviews  with 
experts  in  the  application  area  and  reviews  of  comparable  automated  systems  provide  detailed 
information  requirements  and  usage.  These  requirements  are  driven  into  a detailed  information  model 
which  is  called  an  Application  Reference  Model  (ARM).  Appropriate  segments  of  the  existing  STEP 
integrated  resource  models  are  identified  and  interpreted  to  satisfy  the  application  requirements  as 
specified  in  the  ARM  [Danner92].  This  interpretation  process  results  in  an  Application  interpreted  Model 
(AIM).  The  AIM  supports  the  requirements  of  the  ARM  but  is  based  on  the  information  structures  from 
the  integrated  resource  models.  The  ARM  is  documented  in  one  of  the  accepted  information  modeling 
formats  and  the  AIM  is  to  be  provided  in  two  formats.  Express  and  Express-G  [Palmer91,  iS01 1].  The 
following  outputs  are  produced  from  this  activity: 

• the  ARM,  documented  in  terms  familiar  to  an  application  domain  expert, 

• a formal  and  documented  specification  of  the  AIM  in  Express,  and 

• a graphical  representation  of  the  AIM  in  Express-G. 

The  Interpretation  process  produces  an  application  model  that  is  specific  to  an  application  area  and  also 
consistent  with  other  phases  in  a product  life  cycle.  Both  the  ARM  and  the  AIM  are  validated.  The  ARM 
validation  ensures  that  requirements  are  valid  and  that  the  model  can  support  them.  The  AIM  validation 
ensures  that  the  interpretation  of  the  integrated  resources  is  correct  and  that  the  interpreted  resources 
can  support  the  requirements. 

lii.  Test  Definition 

The  result  of  the  "Test  Definition"  activity  is  a plan  for  validating  the  application  model.  This  information  is 
informative  and  guides  the  testing  process  but  it  is  not  computer-processible.  The  test  plan  describes 
how  typical  users  and  systems  within  an  application  area  use  information  to  perform  the  activities 
described  in  the  application  activity  model.  Results  from  expert  interviews  and  automated  system  reviews 
are  synthesized  into  significant  combinations  of  information  that  identify  non-redundant  and  realistic  test 
conditions,  called  test  purposes,  which  are  based  on  the  usage  requirements.  Each  test  purpose  is  a 
data  access  request  that  needs  to  be  satisfied  using  representative  data  during  the  validation  process. 
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Included  in  this  activity  is  an  identification  of  the  types  of  and  sources  for  data  needed  to  conduct  the 
tests.  The  test  plan  provides  the  organization  to  manage  the  complexity  of  the  required  tests. 

There  are  three  steps  involved  in  the  Test  Definition"  activity.  In  the  test  planning  step,  what  needs  to  be 
tested  is  decided.  A test  plan  with  test  purposes  for  data  usage  is  produced  along  with  a product  profile  ' 
which  describes  unique  characteristics  of  the  product  information.  The  product  profile  is  used  for 
gathering  representative  test  data.  In  the  next  two  steps,  "create  cross  reference  map"  and  "coverage 
analysis",  additional  test  details  are  defined  and  industry  contributed  product  data  is  gathered  and 
organized.  Separate  activities  are  not  defined  for  these  two  steps  because  they  do  not  generate  new 
requirements  for  software  tools,  but  they  are  critical  steps  in  validating  an  application  model. 

The  Test  Definition  activity  produces  two  additional  outputs  which  are  used  by  the  next  activity:  1)  a 
cross  reference  map,  and  2)  a coverage  feedback  report  The  cross  reference  map  indicates  the 
correspondence  between  the  application  model  and  the  representative  test  data  The  creation  of  the 
cross  reference  map  frequently  uncovers  major  structural  flaws  in  the  application  model.  The  coverage 
analysis  of  the  representative  test  data  reveals  unused  segments  of  the  application  model.  If  the  AP 
project  cannot  identify  data  which  corresponds  to  these  segments,  then  the  application  model  needs  to 
change.  Bther  the  model  was  misunderstood  and  requires  clarifying  documentation;  or  additional 
searches  for  corresponding  data  are  necessary.  Ultimately,  the  unused  segments  are  removed  from  the 
application  model  when  their  information  requirements  cannot  be  verified. 

This  activity  produces  four  outputs: 

• an  overall  test  plan  with  usage  test  purposes, 

• a product  profile  and  the  identification  of  representative  test  data  meeting  these  criteria, 

• a cross  reference  map  to  correlate  the  test  data  with  the  application  model,  and 

• a report  which  describes  issues  and  needed  improvements  to  the  application  model. 

IV.  Test  Case  Data  Generation 

During  the  Test  Case  Data  Generation"  activity,  test  case  data  is  assembled  or  built  from  the  contributed 
product  data  which  has  been  selected  by  using  the  characteristics  identified  in  the  product  profile.  The 
objective  is  to  identify  where  in  the  application  model  the  representative  product  data  will  reside  and  if  the 
information  structures  provided  in  the  application  model  can  accommodate  it  Each  piece  of 
industry-contributed  data  should  have  a single,  logical  place  in  the  model.  In  initial  testing  experiences 
[PDES90]  much  of  the  data  was  not  available  in  electronic  form  so  the  test  case  data  was  prepared  by 
hand.  This  was  the  most  labor-intensive  and  error-prone  activity  of  the  entire  process,  but  potentially  the 
most  reusable  for  conformance  testing.  Many  deficiencies  in  the  application  model  are  uncovered  by 
associating  industry  contributed  product  data  with  the  model’s  information  structures. 

The  computer-processible  output  of  this  activity  is  the  test  data.  This  data  directly  supports  the  next 
activity.  Test  Execution  and  Analysis".  The  process  that  makes  that  data  available  for  test  execution 
should  not  need  further  human  intervention.  Also  in  this  activity,  the  test  purposes  for  usage  are  fully 
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detailed  and  documented  as  abstract  test  cases/  This  activity  results  in  the  following  output: 

• detailed  test  data  in  a format  suitable  for  processing  by  the  VTS  software,  e.g.  STEP  exchange  file 

format  [IS021], 

• usage  abstract  test  cases,  and 

• a report  which  describes  issues  and  needed  improvements  to  the  application  model. 

V.  Test  Execution  and  Analysis 

The  Test  Execution  and  Analysis"  activity  involves  the  development  execution,  and  analysis  of  the  test 
cases  against  the  application  model.  In  order  to  execute  the  test  cases,  a computerized  testing 
environment  needs  to  be  established  and  the  test  cases  need  to  be  formally  specified  with  respect  to  the 
testing  environment  Analysis  of  the  test  cases  involves  comparing  the  test  results  to  the  expe^ed  results 
to  determine  the  validity  of  the  application  model.  In  addition,  general  statements  about  whk  any 
Implementation  of  the  AP  must  support  are  documented.  This  activity  produces  the  following  results: 

• validation  test  reports, 

• additional  usage  abstract  test  cases, 

• improved  test  case  data  for  reproducing  test  results, 

• executable  test  cases  for  reproducing  test  results,  and 

• a report  which  describes  issues  and  needed  improvements  to  the  application  model. 

VL  Model  Refinement 

The  "Model  Refinement"  activity  resolves  issues  that  were  uncovered  during  the  testing  process. 
Alternative  solutions  are  proposed  and  the  best  solution  is  selected.  Once  there  is  agreement  on  how  to 
resolve  an  issue,  the  model  is  modified  and  a new  model  is  released  for  validation  testing.  The  process 
of  resolving  an  issue  may  replicate  many  of  the  preceding  steps,  e.g.  the  addition  of  an  entity  for 
resolving  an  issue  might  cause  additional  indust^  contributed  data  to  be  gathered  and  new  test  cases  to 
be  built.  This  activity  results  in  the  following  information: 

• the  refined  application  model  {ARM  orAIMj, 

• an  issue  resolution  statement  describing  the  selected  solution  and  supporting  rationale,  and 

• refined  test  purposes,  abstract  test  cases,  and  executable  test  cases. 

Model  validation  testing  is  an  iterative  process.  The  end  result  of  the  process  is  an  application  model 
suitable  for  inclusion  in  a STEP  AP.  The  model  must  be  both  useful  and  usable  to  be  part  of  the 
standard.  The  involvement  of  a variety  of  application  experts  in  the  validation  process  helps  to  ensure  that 
the  model  is  useful.  There  should  also  be  reviews  by  application  experts  who  were  not  members  of  the 


^ The  abstract  test  case  and  test  purpose  in  conformance  testing  is  a related  concept  but  the  intended 
purpose  is  different.  In  validating  an  application  model,  the  intended  purpose  is  to  evaluate  how  well  the 
model  functions  for  supporting  its  irrtended  scope.  In  conformance  testing,  the  intended  purpose  is  to 
evaluate  if  an  implementation  supports  all  of  the  required  features  of  a standard. 
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AP  project  To  ensure  that  the  model  is  usable,  validation  testing  should  be  repeated  until  the  model 
satisfactorily  supports  the  information  needs  identified  in  the  test  plan. 

The  validation  of  an  application  model  is  dependent  on  the  application  area  under  consideration,  but  the  , 
validation  process  itself  is  constant  and  many  aspects  of  it  can  be  automated  or  supported  by  automation. 
Due  to  the  nature  of  the  standard  being  develops,  it  is  mandatory  that  some  parts  of  the  process  are 
automated.  The  standard  will  enable  the  automatic  sharing  of  data.  Therefore,  the  ability  to  automatically 
access  data  using  the  application  model  needs  to  be  verified.  Section  3 below  discusses  how  this  can  be 
accomplished. 

VIL  Specification  of  Conformance  Requirements 

The  "Specification  of  Conformance  Requirements"  activity  is  performed  when  there  is  confidence  that  the 
application  model  provides  the  functional  capabilities  that  were  specified  in  its  scope.  Conformance 
requirements  specifiy  all  characteristics  that  must  be  satisfied  by  a conforming  implementation  of  the  AP. 
The  three  required  components  for  an  AP  which  fall  within  "Sp^ication  of  Conformance  Requirements" 
are: 

• a conformance  clause  which  specifies  overall  requirements  for  completeness  and  conformance, 

• test  purposes  for  conformance  and  completeness  testing  of  vendor  implementations, 

• Protocol  Inplementation  Conformance  Statement  (PICS)  proforma  which  is  a checklist  for  identifying 

optional  characteristics  that  a vendor  may  claim  to  implement. 

No  further  detail  is  provided  since  there  are  many  unresolved  issues  relating  to  this  topic  (see  Annex  C of 
the  "Guidelines  for  Development  and  Approval  of  STEP  Application  Protocols"  [PalmerSI]  for  further 
enumeration). 


3 Automation  for  the  Validation  Testing  Methodology 

This  section  describes  the  automated  dataflow  within  the  VTS  at  the  National  PDES  Testbed.  The 
software,  which  supports  the  validation  testing  process,  simulates  the  information  access  requirements  for 
the  application  area  being  tested.  The  VTS  software  will  provide  a controlled  environment  for  model 
validation  testing,  thereby  reducing  the  potential  for  introducing  errors  into  the  process.  The  VTS 
software  and  the  control  of  the  supporting  environment  will  also  reduce  the  level  of  computer 
sophistication  and  interaction  needed  so  that  the  users  of  the  system  will  be  able  to  concenti’ate  on 
validating  the  application  models. 

The  primary  requirement  of  this  process  is  the  capability  to  manipulate  and  represent  an  application 
model  and  associated  data  for  a variety  of  purposes.  Therefore,  many  functional  requirements 
{Morris91a]  such  as  the  ability  to  display  and  access  the  contents  of  models  written  in  Express  [IS01 1] 
and  the  ability  to  manage  the  versions  of  documents  and  other  files,  are  common  among  various 
activities.  Some  of  these  requirements  such  as  word  processing  for  preparation  of  documents,  database 
access  and  persistent  storage,  and  computer-aided  design  analysis  of  geometry,  are  not  unique  to  STEP 
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and  are  available  in  commercial  systems.  For  these  requirements,  commercial  systems  will  be  used  and 
integrated  with  the  VTS  software.  Other  requirements  that  are  unique  to  STEP  are  either  available  from 
related  projects  or  will  be  developed  for  the  VTS. 

Each  activity  of  the  validation  testing  process  consumes  and  produces  specifications  or  data.  A subset  of 
this  material  is  directly  processible  and  can  be  used  to  automate  the  activities.  This  automation  parallels 
the  flow  of  information  between  the  activities  described  in  Section  2.  Table  1 illustrates  information  inputs 
and  outputs;  the  entries  in  bold  represent  the  computer-processibie  portion  of  the  information  flow 
between  the  activities.  The  remainder  of  this  section  focuses  on  only  those  portions  which  are  currently 
computer-processibie.  For  a more  general  discussion  of  the  information  flow  for  model  validation,  see 
Mitchell  [Mitch91]. 
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INPUT 

OUTPUT 

Scoping  the  AppOcation 
Context 

AppOcation  Requirements 

Scope  & Requrements  Statement 

Application  Acflvlty  Modal 

Model  Construction 

Scope  & Requirements  Statement 

Application  Aetivfty  Model 

Inttgrated  Raaourea  Modala 

Application  Modala  (La.  ARM,  AIM} 

Graphic  Information  Model 

Test  Definition 

AppOcation  RequOaments 

Sc^  & Requ^ements  Statement 

Application  Activity  Model 

Application  Model  In  Exprau 

Test  Plan  with  Test  Purposes 

Product  Profile 

Contributed  Product  Data 

Cross  Reference  Map 

Model  Issues 

Test  Case  Data  Generation 

Scope  & Requirements  Statement 

Application  Aetivfty  Modal 

Application  Modal  In  Expraaa 

Product  Profile 

Test  Plan  with  Test  Purposes 

Contributed  Product  Data  (].e.  IQES  fUas) 
Cross-Reference  Map 

Teat  Caaa  Data  (La.  STEP  IDaa) 

Abstract  Test  Caks 

Model  Issues 

Test  Execution  and 

Analysis 

Scope  & Requirements  Statement 

Application  Activity  Modal 

Application  Modal  In  Expraaa 

Test  Plan  with  Test  Purpc^ 

Abstract  Test  Cases 

Teat  Caaa  Data  (La.  STEP  filaa) 

Exacutabia  Teat  Cases 

Validation  Report 

Model  issues 

Refined  Abstract  Test  Cases  & Teat  Caaa 

Data 

Tool  Enhancement  Requirements 

Model  Refinement 

Model  Issues 

Application  Model  In  Express 

Application  Modal  in  Express 

Model  Log  with  Resolutions 

Conformance  Requirements 

Application  Modal  In  Expraaa 

Abstract  Test  Cases 

Test  Case  Data 

Conformance  Clause 

Abstract  Test  Suite 

Model  Issues 

IBin  intormation  Plow  Between  Model  Development  and  validation  Acttvntes 


Rgura  1 below,  illustrates  the  relationship  between  the  activities  and  the  currently  computer-processible 
information  flow. 


The  Model  Construction  activity  produces  application  models  in  both  human  and  computer-interpretable 
formats.  Currently  only  the  B^ress  version  of  the  application  model  is  directly  used  as  a basis  for  the 
software.  The  STEP  integrated  resource  models  are  also  represented  in  Express  and  supply  a basis  for 
the  application  model  being  developed  and  validated.  If  additional  computer-processible  representations 
for  these  outputs  were  available,  such  as  a test  notation  for  abstract  test  cases,  the  amount  of  automation 
coukJ  be  inaeased. 
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The  Test  Definition  activity  involves  a great  deal  of  human  interaction,  synthesis,  and  analysis.  It  Is  the 
least  automatable  activity  in  the  process.  The  automation  is  limited  to  assistance  in  referencing  the 
application  and  activity  models  and  in  the  preparation  of  documentation.  The  work  being  done  on 
conformance  testing  by  the  developers  of  STEP  includes  a formal  test  notation  language  that  may 
increase  the  ability  to  provide  additional  automated  assistance. 

The  primary  automation  for  the  Test  Case  Data  Generation  activity  is  for  assistance  in  preparing  test 
data.  The  industry  contributed  data  is  represented  in  many  formats,  but  to  be  usable  by  validation 
testing,  the  test  case  data  must  be  formed  into  a STEP  exchange  file  format  [IS021]  or  loaded  into  a 
database  system  which  has  been  built  to  manange  STEP  data  structures.  A reliable  and  efficient  way  to 
receive  a limited  portion  of  this  data,  principally  geometric  entities,  is  in  the  form  of  an  Initial  Graphics 
Exchange  Specification  (IGES)  [IP091]  file  extracted  from  a CAD  system.  IGES  files  can  be  translated  to 
STEP  exchange  files  [BreeseSI].  Additional  contributed  data  needs  to  be  prepared  manually  to  complete 
the  information  required  for  the  application  model. 

In  the  Test  Execution  and  Analysis  activity,  executable  test  cases  are  generated,  executed,  and  the 
results  analyzed.  This  activity  allows  for  a high  degree  of  automation.  The  typical  testing  scenario 
follows; 

1.  the  application  modei  and  test  data  are  represented  in  a database; 

2.  executable  tests  are  specified  in  the  database  system's  query  language; 

3.  the  queries  which  simulate  typical  application  data  access  requirements  are  executed;  and 

4.  the  results  are  analyzed,  issues  against  the  application  model  are  documented,  and  a validation  report 
is  generated. 

The  Model  Refinement  scW\ty  leads  to  a new  application  model  and  may  contribute  to  refinements  to  the 
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STEP  integrated  resource  models.  Throughout  the  validation  testing  process,  any  deficiencies  in  the 
application  model,  the  test  cases,  or  the  test  environment  are  documented,  and  appropriate 
enhancements  are  made.  Appropriate  steps  in  the  validation  process  are  repeat^  using  the  refined 
application  model  and  test  case  data  for  any  test  purpose  that  was  affected  by  these  refinements. 

The  Conformance  Requirements  activity  is  still  evolving.  The  current  understanding  of  what  should  be 
accomplished  in  this  activity  is  described  in  Palmer  [Palmer91].  There  is  potential  for  reusing  test  case 
data,  abstract  test  cases  and  VTS  tools  for  these  purposes.  The  Intended  purpose  of  this  activity  is  to 
specify  all  of  the  requirements  that  a vendor  implementation  of  an  AP  must  satisfy.  The  VTS  project 
efforts  will  evaluate  these  requirements  when  they  become  available. 

4 Model  Validation  Testing  at  the  National  PDES  Testbed 

The  National  PDES  Testbed  has  been  used  for  the  validation  testing  of  STEP  application  models  since 
1989.  The  software  currently  in  place  in  the  National  PDES  Testbed  [6reese91]  provides  some  of  the 
automation  desired.  An  important  point  to  make  is  that  ail  of  the  software  is  being  developed  to  generate 
a test  enviroment  for  the  schema  under  test.  While  the  current  software  will  support  any  data  model  that 
has  been  developed  in  Express,  knowledge  of  the  other  data  modeling  techniques  used  within  STEP  has 
contributed  to  the  design.  This  section  summarizes  the  direction  for  future  improvements  and  additions  to 
the  validation  testing  software  at  the  Testbed.  This  direction  is  based  on  past  experiences  with  the 
software  and  STEP  development  methods,  which  helped  clarify  the  needs  for  the  validation  testing 
software. 

4.1  Experiences  with  the  Interim  System 

The  interim  software  employed  in  the  validation  testing  of  application  models  consists  of  a set  of 
Independent  tools  which  operate  in  a variety  of  computer  environments.  The  current  method  of  sharing 
data  in  the  testing  process  is  by  exchanging  data  files  between  these  tools.  This  requires  data  translation 
and  manual  intervention,  which  introduces  the  potential  for  errors  and  inconsistencies,  every  time  data  is 
processed  in  the  testing  activities.  Moreover,  the  process  of  importing  and  exporting  between  tools  is 
time-consuming. 

The  automation  for  this  testing  process  is  currently  provided  by  software  tools  which  translate  the 
application  model  and  the  test  data  among  a number  of  formats  [Clark90a,PDES91].  The  software 
includes: 

• an  Express  compiler  and  translators  for  representing  the  application  model; 

• an  editor  for  STEP  data  [ClarkSOb]  which  structures  the  information  as  specified  in  Express  for 
the  application  model; 

• a relational  database  [Date90]  which  provides  data  access  and  storage  management  along  with 
a query  capability; 

• a STEP  exchange  file  parser  and  loaders  for  populating  the  STEP  editor  and  database; 

• Export  facilities  for  extracting  data  from  the  editor  or  database  into  STEP  exchange  files; 

• an  IGES  to  STEP  translator  for  converting  digital  product  data  from  a Computer-Aided  Design 
system  to  STEP;  and 
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• a visuaiizer  of  geometric  model  data  for  display  of  a very  limited  set  of  STEP  data. 

The  interim  system  has  some  of  the  needed  functionality  but  provides  unacceptable  performance  for 
some  of  the  functions.  The  most  significant  improvements  to  the  current  system  can  be  made  by 
replacing  the  STEP  editor  and  database  management  system  with  improv^  and  integrated  components. 
Both  tools  have  suffered  from  significant  performance  problems  with  the  large  sets  of  complex  data 
typical  of  engineering  uses  [PDES90]. 

In  addition  the  STEP  editor  and  database  system  currently  use  different  data  representation  paradigms  to 
represent  the  application  model  and  its  associated  data.  This  places  a burden  on  the  users  to 
uriderstand  the  different  representational  formats  and  the  relationship  between  these  formats. 

4.2  Future  Directions  for  the  Model  Validation  Software 

The  software  needs  for  the  STEP  AP  model  validation  process  can  be  divided  into  two  categories: 

1.  automation  of  the  validation  process  by  simulating  the  data  access  requirements  of  the  application; 
and 

2.  automation  to  support  the  validation  process,  through  assistance  for  preparing  documentation  and  for 
referencing  and  browsing  the  application  or  integrated  resource  models. 

The  first  category  is  a mandatory  requirement  for  effective  validation  testing.  The  tests  resulting  from  the 
process  must  be  computer-processible  and  repeatable.  These  tests  reflect  the  intended  usage  of  the 
application  model.  Software  for  this  purpose  is  the  first  priority  for  future  implementation  efforts. 

The  second  category  is  partially  met  in  the  current  system  by  word  processing  and  drawing  packages. 
These  solutions  provide  limited  support  for  these  functions  and  leading  to  inefficient  use  of  human 
resources.  However,  these  automation  needs  will  not  be  addressed  until  the  first  category  is  supported. 

Two  tools  are  most  important  for  supporting  the  first  category  of  automation,  the  simulation  of  data  access 
based  upon  application  information  requirements: 

1.  a STEP  data  editor,  and 

2.  a database  system  with  a query  capability. 

Current  efforts  for  the  VTS  software  focus  on  developing  an  improved  and  integrated  STEP  data  editor. 
This  editor  is  being  developed  so  that  it  will  integrate  with  a database  system.  However,  the  editor  will 
not  depend  on  having  a database  system.  The  VTS  software  will  provide  an  integrated  set  of  functions 
which  will  provide  a more  effective  and  efficient  environment  and  one  more  capable  of  simulating  the  data 
access  needs  of  the  application  area.  In  summary  the  VTS  software  will  make  improvements  over  the 
interim  software  in  the  following  areas: 

• performance,  in  terms  of  both  computation  time  and  reliability; 

• workflow  automation  to  eliminate  manual  intervention  where  possible; 

• more  sophisticated  support  for  editing  of  STEP  data  to  reduce  inconsistences  in  the  data; 
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• error  checking  to  reduce  the  potential  for  errors  and  improve  error  detection; 

• expansion  of  functionality  to  address  the  needs  of  model  scoping,  model  construction,  and 
model  refinement; 

• provision  of  a single  interface  to  the  software,  which  will  reduce  the  effort  needed  to  learn  the  * 
tools;  and 

• adoption  of  a data  representation  paradigm  that  resembles  Express  more  dosely  than  the 
relational  paradigm. 

When  completed  and  tested,  the  VIS  software  and  supporting  documentation  will  be  made  available 
through  the  National  PDES  Testbed  STEP  On-line  Information  System  [Katz91].  as  are  the  NIST 
developed  portions  of  the  interim  system.’ 

4,3  Future  Directions  for  AP  Validation  Methods 

There  are  two  areas  where  additional  effort  in  defining  validation  methods  are  needed.  These  are: 

1.  validation  techniques  for  the  AP  Conformance  Testing  requirements;  and 

2.  definition  of  the  relationship  between  AP  model  validation  testing  and  the  specification  of  AP 
conformance  tests. 

Summary 

The  validation  testing  of  data  models  is  needed  to  ensure  that  the  technical  solutions  provided  by 
integrated  model  will  work  in  a practical  sense.  An  Application  Protocol  (AP)  within  STEP  is  a 
specification  of  data  sharing  requirements  for  a particular  application  area.  Application  Protocols  are 
designed  to  permit  practical  implementations  of  STEP.  The  model  validation  focuses  on  the  principal 
mechanism  for  specifying  the  data  sharing  requirements,  an  application  specific  data  model.  The  body  of 
the  paper  describes  the  process  by  which  an  application  model  is  validate. 

Application  model  development  and  validation  is  a complex  process  that  relies  extensively  on  human 
ca^ilities  for  analysis,  judgement  and  synthesis  of  large  amounts  of  diverse  information.  This  process 
requires  the  support  from  automation  to  produce  a technically  complete  AP.  The  model  validation 
process  and  the  STEP  development  methods  place  unique  requirements  on  the  software  that  will  be 
needed  to  support  the  effective  testing  of  STEP.  The  National  POES  Testbed  at  the  National  Institute  of 
Standards  and  Technology  has  undertaken  a software  project  to  support  this  process.  The  current 
direction  of  this  project  has  been  formulated  from  our  initial  experiences  in  exercising  this  process  and 
with  software  automation  for  the  model  validation  testing.  In  addition,  this  paper  introduces  the  potential 
contribution  that  application  model  validation  and  validation  tools  could  make  to  the  conformance  testing 
of  AP  implementations. 
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